Copyto: fail on error and don't overwrite files

doyoya · September 13, 2024, 9:29am

What is the problem you are having with rclone?

I want to transfer a single file from S3 to S3, using to following rules:

don't overwrite existing files in the destination
fail (status code != 0) on errors (e.g. network issues, file not existing in the source, etc.)

Elaboration:

I've noticed, that, when transferring from S3 to S3, when the file in the source is missing, and without the --error-on-no-transfer flag, rclone exits with code 0, basically saying, everything is ok. Which is kind of unexpected, because, it's not ok, it's a failure scenario. We had faulty configurations running (it can happen, we are all humens after all ), and never saw it, only wondered why files were missing). That's why I added the --error-on-no-transfer in the first place.
When the file already exists in the destination (which can happen sometimes), and with --error-on-no-transfer set, rclone now exists with code 9 (I think). Which is kind of understandable, because of the --error-on-no-transfer flag.

However, that leaves me to a kind-of unsolvable problem, even if it's a simple scenario. I can either:

... use just --ignore-existing, and I can't be sure if a file was actually transferred or not (rclone exists with code 0, not indicating any problem) (bad)
... use just --error-on-no-transfer and overwrite files in the destination S3 bucket, which could mess up data integrity (bad)
... use --error-on-no-transfer and --ignore-existing and being unable to distinguish between "file-already-exists-in-destination-bucket", "file-is-missing-in-the-source-bucket", and probably(?) a few other error scenarios? (rclone exists with code 9 in at least those 2 scenarios) (also bad)

It looks like I only have bad options here. I somehow need rclone to exit with code 0 when the file wasn't transferred because it already exists in the destination. Does anyone know, how to solve this? Is there a parameter, that I missed, which will make it work?

Run the command 'rclone version' and share the full output of the command.

rclone v1.67.0
- os/version: alpine 3.20.0 (64 bit)
- os/kernel: 5.14.0-362.18.1.el9_3.x86_64 (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.22.4
- go/linking: static
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

S3 (both, source and destination)

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

rclone copyto --error-on-no-transfer --ignore-existing --s3-no-check-bucket --verbose "s3-src:[...]" "s3-dst:[...]"

Please run 'rclone config redacted' and share the full output. If you get

skipped (because, I think it's not important for this issue)

A log from the command that you were trying to run with the `-vv` flag