What is the problem you are having with rclone?
Same problem as described here.
Same solution works but this solution has some problems in my case (read on).
Run the command 'rclone version' and share the full output of the command.
rclone v1.69.2
- os/version: ubuntu 18.04 (64 bit)
- os/kernel: 4.15.0-213-generic (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.24.2
- go/linking: static
- go/tags: none
Which cloud storage system are you using? (eg Google Drive)
AWS S3
The command you were trying to run (eg rclone copy /tmp remote:tmp
)
rclone copy --s3-no-check-bucket test.txt s3:/my-bucket/
Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.
[s3]
type = s3
env_auth = true
region = us-west-2
provider = AWS
acl = private
location_constraint = us-west-2
### Double check the config for sensitive info before posting publicly
A log from the command that you were trying to run with the -vv
flag
2025/05/02 17:52:32 DEBUG : rclone: Version "v1.69.2" starting with parameters ["rclone" "-vv" "copy" "--s3-no-check-bucket" "test.txt" "s3:/my-bucket/"]
2025/05/02 17:52:32 DEBUG : Creating backend with remote "test.txt"
2025/05/02 17:52:32 DEBUG : Using config file from "/home/dtenenba/.config/rclone/rclone.conf"
2025/05/02 17:52:32 DEBUG : fs cache: renaming child cache item "test.txt" to be canonical for parent "/home/dtenenba"
2025/05/02 17:52:32 DEBUG : Creating backend with remote "s3:/my-bucket/"
2025/05/02 17:52:32 DEBUG : s3: detected overridden config - adding "{Dn7qA}" suffix to name
2025/05/02 17:52:32 DEBUG : fs cache: renaming cache item "s3:/my-bucket/" to be canonical "s3{Dn7qA}:my-bucket"
2025/05/02 17:52:32 DEBUG : test.txt: Need to transfer - File not found at Destination
2025/05/02 17:52:32 DEBUG : test.txt: md5 = e19c1283c925b3206685ff522acfe3e6 (Local file system at /home/dtenenba)
2025/05/02 17:52:32 DEBUG : test.txt: md5 = c0a79ef5f7f4521da78c0482d120ec7b (S3 bucket my-bucket)
2025/05/02 17:52:32 ERROR : test.txt: corrupted on transfer: md5 hashes differ src(Local file system at /home/dtenenba) "e19c1283c925b3206685ff522acfe3e6" vs dst(S3 bucket my-bucket) "c0a79ef5f7f4521da78c0482d120ec7b"
2025/05/02 17:52:32 INFO : test.txt: Removing failed copy
2025/05/02 17:52:32 ERROR : Attempt 1/3 failed with 1 errors and: corrupted on transfer: md5 hashes differ src(Local file system at /home/dtenenba) "e19c1283c925b3206685ff522acfe3e6" vs dst(S3 bucket my-bucket) "c0a79ef5f7f4521da78c0482d120ec7b"
2025/05/02 17:52:32 DEBUG : test.txt: Need to transfer - File not found at Destination
2025/05/02 17:52:32 DEBUG : test.txt: md5 = e19c1283c925b3206685ff522acfe3e6 (Local file system at /home/dtenenba)
2025/05/02 17:52:32 DEBUG : test.txt: md5 = 7618e7c55d86b4382fd31d3db8489781 (S3 bucket my-bucket)
2025/05/02 17:52:32 ERROR : test.txt: corrupted on transfer: md5 hashes differ src(Local file system at /home/dtenenba) "e19c1283c925b3206685ff522acfe3e6" vs dst(S3 bucket my-bucket) "7618e7c55d86b4382fd31d3db8489781"
2025/05/02 17:52:32 INFO : test.txt: Removing failed copy
2025/05/02 17:52:32 ERROR : Attempt 2/3 failed with 1 errors and: corrupted on transfer: md5 hashes differ src(Local file system at /home/dtenenba) "e19c1283c925b3206685ff522acfe3e6" vs dst(S3 bucket my-bucket) "7618e7c55d86b4382fd31d3db8489781"
2025/05/02 17:52:32 DEBUG : test.txt: Need to transfer - File not found at Destination
2025/05/02 17:52:32 DEBUG : test.txt: md5 = e19c1283c925b3206685ff522acfe3e6 (Local file system at /home/dtenenba)
2025/05/02 17:52:32 DEBUG : test.txt: md5 = 1bfa954c21e8f72467687acbaac4d8be (S3 bucket my-bucket)
2025/05/02 17:52:32 ERROR : test.txt: corrupted on transfer: md5 hashes differ src(Local file system at /home/dtenenba) "e19c1283c925b3206685ff522acfe3e6" vs dst(S3 bucket my-bucket) "1bfa954c21e8f72467687acbaac4d8be"
2025/05/02 17:52:32 INFO : test.txt: Removing failed copy
2025/05/02 17:52:32 ERROR : Attempt 3/3 failed with 1 errors and: corrupted on transfer: md5 hashes differ src(Local file system at /home/dtenenba) "e19c1283c925b3206685ff522acfe3e6" vs dst(S3 bucket my-bucket) "1bfa954c21e8f72467687acbaac4d8be"
2025/05/02 17:52:32 INFO :
Transferred: 45 B / 45 B, 100%, 0 B/s, ETA -
Errors: 1 (retrying may help)
Elapsed time: 0.3s
2025/05/02 17:52:32 DEBUG : 6 go routines active
2025/05/02 17:52:32 NOTICE: Failed to copy: corrupted on transfer: md5 hashes differ src(Local file system at /home/dtenenba) "e19c1283c925b3206685ff522acfe3e6" vs dst(S3 bucket my-bucket) "1bfa954c21e8f72467687acbaac4d8be"
The problem
I know this can be solved by adding --s3-server-side-encryption aws:kms
, however we are using rclone in an automated context where we do not know in advance if the bucket we are copying to is encrypted or not.
I also know that adding either of these will cause the transfer to work:
--ignore-checksum
--s3-upload-cutoff=0
Obviously the first is bad and I am not sure of the implications of the second.
Is there anything I need to be aware of if I use that flag for all transfers to S3, regardless of the size of the file(s) to be transferred?
Also, why is this not a problem in the AWS CLI (which does not need to know if a target bucket is encrypted or not) but it affects rclone?
Our automated program could perhaps query the bucket metadata and find out if it is encrypted, but the user may not have permission to do that (a lot of users can't create or list buckets either which is why I have --s3-no-check-bucket
in the command).
TIA