S3 checksum failing for all copies

Help and Support
1

What is the problem you are having with rclone?

All copies to and from S3 are failing the checksum. However, everything works fine when I use the --ignore-checksum flag. I confirmed that the files are uncorrupted when I use rclone copy --ignore-checksum to transfer them to S3 and back.

The S3 bucket is unencrypted.

What is your rclone version (output from rclone version)

rclone v1.53.3

  • os/arch: linux/amd64
  • go version: go1.15.5

Which OS you are using and how many bits (eg Windows 7, 64 bit)

RHEL 7.9, 64 bit

Which cloud storage system are you using? (eg Google Drive)

S3

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copy ./tmp S3:ticfoobar/tmp

The rclone config contents with secrets removed.

[S3]
type = s3
provider = AWS
env_auth = true
region = us-east-1
acl = private
server_side_encryption = aws:kms
sse_kms_key_id = [REDACTED]

A log from the command with the -vv flag

2021/01/05 12:13:31 DEBUG : rclone: Version "v1.53.3" starting with parameters ["rclone" "copy" "-vv" "./tmp" "S3:ticfoobar" "--log-file" "rclone.log"]
2021/01/05 12:13:31 DEBUG : Creating backend with remote "./tmp"
2021/01/05 12:13:31 DEBUG : Using config file from "/home/aostow/.config/rclone/rclone.conf"
2021/01/05 12:13:31 DEBUG : fs cache: renaming cache item "./tmp" to be canonical "/home/aostow/dev/scratch/tmp"
2021/01/05 12:13:31 DEBUG : Creating backend with remote "S3:ticfoobar"
2021/01/05 12:13:31 DEBUG : S3 bucket ticfoobar: Waiting for checks to finish
2021/01/05 12:13:31 DEBUG : S3 bucket ticfoobar: Waiting for transfers to finish
2021/01/05 12:13:31 DEBUG : 10.txt: MD5 = d12dd13f4dfe9adef553f0490763d4df (Local file system at /home/aostow/dev/scratch/tmp)
2021/01/05 12:13:31 DEBUG : 10.txt: MD5 = 13967b8be7d06dd256abd77f77266899 (S3 bucket ticfoobar)
2021/01/05 12:13:31 ERROR : 10.txt: corrupted on transfer: MD5 hash differ "d12dd13f4dfe9adef553f0490763d4df" vs "13967b8be7d06dd256abd77f77266899"
2021/01/05 12:13:31 INFO  : 10.txt: Removing failed copy
2021/01/05 12:13:31 DEBUG : 2.txt: MD5 = f303b7d2f2b87f9e16df05e2bca7c409 (Local file system at /home/aostow/dev/scratch/tmp)
2021/01/05 12:13:31 DEBUG : 2.txt: MD5 = a1a8ed20865a054c9176e3931865690c (S3 bucket ticfoobar)
2021/01/05 12:13:31 ERROR : 2.txt: corrupted on transfer: MD5 hash differ "f303b7d2f2b87f9e16df05e2bca7c409" vs "a1a8ed20865a054c9176e3931865690c"
2021/01/05 12:13:31 INFO  : 2.txt: Removing failed copy
2021/01/05 12:13:31 DEBUG : 4.txt: MD5 = fe59a05ba880dc4ad0d50e7401b1648d (Local file system at /home/aostow/dev/scratch/tmp)
2021/01/05 12:13:31 DEBUG : 4.txt: MD5 = d197b61806a583234266966321de9778 (S3 bucket ticfoobar)
2021/01/05 12:13:31 ERROR : 4.txt: corrupted on transfer: MD5 hash differ "fe59a05ba880dc4ad0d50e7401b1648d" vs "d197b61806a583234266966321de9778"
2021/01/05 12:13:31 INFO  : 4.txt: Removing failed copy
2021/01/05 12:13:31 DEBUG : 3.txt: MD5 = f2b33fb7b3d0eb95090a16060e6a24f9 (Local file system at /home/aostow/dev/scratch/tmp)
2021/01/05 12:13:31 DEBUG : 3.txt: MD5 = e9525dc05a197f741796bd33d2c6785c (S3 bucket ticfoobar)
2021/01/05 12:13:31 ERROR : 3.txt: corrupted on transfer: MD5 hash differ "f2b33fb7b3d0eb95090a16060e6a24f9" vs "e9525dc05a197f741796bd33d2c6785c"
2021/01/05 12:13:31 INFO  : 3.txt: Removing failed copy
2021/01/05 12:13:31 DEBUG : 1.txt: MD5 = b026324c6904b2a9cb4b88d6d61c81d1 (Local file system at /home/aostow/dev/scratch/tmp)
2021/01/05 12:13:31 DEBUG : 1.txt: MD5 = ec9031ec99a292d622d6ec5f0af63a7b (S3 bucket ticfoobar)
2021/01/05 12:13:31 ERROR : 1.txt: corrupted on transfer: MD5 hash differ "b026324c6904b2a9cb4b88d6d61c81d1" vs "ec9031ec99a292d622d6ec5f0af63a7b"
2021/01/05 12:13:31 INFO  : 1.txt: Removing failed copy
2021/01/05 12:13:31 DEBUG : 6.txt: MD5 = 14ba5568c6103ac56facf99277adb053 (Local file system at /home/aostow/dev/scratch/tmp)
2021/01/05 12:13:31 DEBUG : 6.txt: MD5 = ee17632fe83e3ca127d0ba2d68b7cbee (S3 bucket ticfoobar)
2021/01/05 12:13:31 ERROR : 6.txt: corrupted on transfer: MD5 hash differ "14ba5568c6103ac56facf99277adb053" vs "ee17632fe83e3ca127d0ba2d68b7cbee"
2021/01/05 12:13:31 INFO  : 6.txt: Removing failed copy
2021/01/05 12:13:31 DEBUG : 5.txt: MD5 = 9f96ef92fe742165873c313662f1f2b8 (Local file system at /home/aostow/dev/scratch/tmp)
2021/01/05 12:13:31 DEBUG : 5.txt: MD5 = 19e641e3da2a6b3aaa0e0564843d3a50 (S3 bucket ticfoobar)
2021/01/05 12:13:31 ERROR : 5.txt: corrupted on transfer: MD5 hash differ "9f96ef92fe742165873c313662f1f2b8" vs "19e641e3da2a6b3aaa0e0564843d3a50"
2021/01/05 12:13:31 INFO  : 5.txt: Removing failed copy
2021/01/05 12:13:31 DEBUG : 7.txt: MD5 = 5d336a7898bfda02ffc516e3b35a7230 (Local file system at /home/aostow/dev/scratch/tmp)
2021/01/05 12:13:31 DEBUG : 7.txt: MD5 = f87e9f4c0c9fac40f1df2a1710abe577 (S3 bucket ticfoobar)
2021/01/05 12:13:31 ERROR : 7.txt: corrupted on transfer: MD5 hash differ "5d336a7898bfda02ffc516e3b35a7230" vs "f87e9f4c0c9fac40f1df2a1710abe577"
2021/01/05 12:13:31 INFO  : 7.txt: Removing failed copy
2021/01/05 12:13:31 DEBUG : 8.txt: MD5 = 5c38c9d77fee15fd82d3ab04b199930b (Local file system at /home/aostow/dev/scratch/tmp)
2021/01/05 12:13:31 DEBUG : 8.txt: MD5 = d3a39d52108fc9336a001fc230946e44 (S3 bucket ticfoobar)
2021/01/05 12:13:31 ERROR : 8.txt: corrupted on transfer: MD5 hash differ "5c38c9d77fee15fd82d3ab04b199930b" vs "d3a39d52108fc9336a001fc230946e44"
2021/01/05 12:13:31 INFO  : 8.txt: Removing failed copy
2021/01/05 12:13:31 DEBUG : 9.txt: MD5 = 12808f4d90a07ec63c617ac251dd80bf (Local file system at /home/aostow/dev/scratch/tmp)
2021/01/05 12:13:31 DEBUG : 9.txt: MD5 = bf03b53e0b28da20a4cc7bd976390a36 (S3 bucket ticfoobar)
2021/01/05 12:13:31 ERROR : 9.txt: corrupted on transfer: MD5 hash differ "12808f4d90a07ec63c617ac251dd80bf" vs "bf03b53e0b28da20a4cc7bd976390a36"
2021/01/05 12:13:31 INFO  : 9.txt: Removing failed copy
2021/01/05 12:13:31 INFO  : There was nothing to transfer
2021/01/05 12:13:31 ERROR : Attempt 1/3 failed with 10 errors and: corrupted on transfer: MD5 hash differ "12808f4d90a07ec63c617ac251dd80bf" vs "bf03b53e0b28da20a4cc7bd976390a36"
2021/01/05 12:13:31 DEBUG : S3 bucket ticfoobar: Waiting for checks to finish
2021/01/05 12:13:31 DEBUG : S3 bucket ticfoobar: Waiting for transfers to finish
2021/01/05 12:13:31 DEBUG : 2.txt: MD5 = f303b7d2f2b87f9e16df05e2bca7c409 (Local file system at /home/aostow/dev/scratch/tmp)
2021/01/05 12:13:31 DEBUG : 2.txt: MD5 = 2d474e91b0bae61c3a6d83e207453c5e (S3 bucket ticfoobar)
2021/01/05 12:13:31 ERROR : 2.txt: corrupted on transfer: MD5 hash differ "f303b7d2f2b87f9e16df05e2bca7c409" vs "2d474e91b0bae61c3a6d83e207453c5e"
2021/01/05 12:13:31 INFO  : 2.txt: Removing failed copy
2021/01/05 12:13:32 DEBUG : 1.txt: MD5 = b026324c6904b2a9cb4b88d6d61c81d1 (Local file system at /home/aostow/dev/scratch/tmp)
2021/01/05 12:13:32 DEBUG : 1.txt: MD5 = 83f622b11a9402413e865386b74cc21f (S3 bucket ticfoobar)
2021/01/05 12:13:32 ERROR : 1.txt: corrupted on transfer: MD5 hash differ "b026324c6904b2a9cb4b88d6d61c81d1" vs "83f622b11a9402413e865386b74cc21f"
2021/01/05 12:13:32 INFO  : 1.txt: Removing failed copy
2021/01/05 12:13:32 DEBUG : 3.txt: MD5 = f2b33fb7b3d0eb95090a16060e6a24f9 (Local file system at /home/aostow/dev/scratch/tmp)
2021/01/05 12:13:32 DEBUG : 3.txt: MD5 = 4160a4f7972d9706dc6b0ecd8f185b16 (S3 bucket ticfoobar)
2021/01/05 12:13:32 ERROR : 3.txt: corrupted on transfer: MD5 hash differ "f2b33fb7b3d0eb95090a16060e6a24f9" vs "4160a4f7972d9706dc6b0ecd8f185b16"
2021/01/05 12:13:32 INFO  : 3.txt: Removing failed copy
2021/01/05 12:13:32 DEBUG : 10.txt: MD5 = d12dd13f4dfe9adef553f0490763d4df (Local file system at /home/aostow/dev/scratch/tmp)
2021/01/05 12:13:32 DEBUG : 10.txt: MD5 = 416489b2805d197b0b19a8e63b6cde9a (S3 bucket ticfoobar)
2021/01/05 12:13:32 ERROR : 10.txt: corrupted on transfer: MD5 hash differ "d12dd13f4dfe9adef553f0490763d4df" vs "416489b2805d197b0b19a8e63b6cde9a"
2021/01/05 12:13:32 INFO  : 10.txt: Removing failed copy
2021/01/05 12:13:32 DEBUG : 6.txt: MD5 = 14ba5568c6103ac56facf99277adb053 (Local file system at /home/aostow/dev/scratch/tmp)
2021/01/05 12:13:32 DEBUG : 6.txt: MD5 = 144e7d0475ee3ce8be44ae52bf1642b0 (S3 bucket ticfoobar)
2021/01/05 12:13:32 ERROR : 6.txt: corrupted on transfer: MD5 hash differ "14ba5568c6103ac56facf99277adb053" vs "144e7d0475ee3ce8be44ae52bf1642b0"
2021/01/05 12:13:32 INFO  : 6.txt: Removing failed copy
2021/01/05 12:13:32 DEBUG : 5.txt: MD5 = 9f96ef92fe742165873c313662f1f2b8 (Local file system at /home/aostow/dev/scratch/tmp)
2021/01/05 12:13:32 DEBUG : 5.txt: MD5 = 280b54f84f89fb05c2db3aa88ba8de37 (S3 bucket ticfoobar)
2021/01/05 12:13:32 ERROR : 5.txt: corrupted on transfer: MD5 hash differ "9f96ef92fe742165873c313662f1f2b8" vs "280b54f84f89fb05c2db3aa88ba8de37"
2021/01/05 12:13:32 INFO  : 5.txt: Removing failed copy
2021/01/05 12:13:32 DEBUG : 4.txt: MD5 = fe59a05ba880dc4ad0d50e7401b1648d (Local file system at /home/aostow/dev/scratch/tmp)
2021/01/05 12:13:32 DEBUG : 4.txt: MD5 = 85ea4c5db05c9d28179881a1ad385113 (S3 bucket ticfoobar)
2021/01/05 12:13:32 ERROR : 4.txt: corrupted on transfer: MD5 hash differ "fe59a05ba880dc4ad0d50e7401b1648d" vs "85ea4c5db05c9d28179881a1ad385113"
2021/01/05 12:13:32 INFO  : 4.txt: Removing failed copy
2021/01/05 12:13:32 DEBUG : 7.txt: MD5 = 5d336a7898bfda02ffc516e3b35a7230 (Local file system at /home/aostow/dev/scratch/tmp)
2021/01/05 12:13:32 DEBUG : 7.txt: MD5 = 7e1b94adff34ce9b5c4622f9a810f035 (S3 bucket ticfoobar)
2021/01/05 12:13:32 ERROR : 7.txt: corrupted on transfer: MD5 hash differ "5d336a7898bfda02ffc516e3b35a7230" vs "7e1b94adff34ce9b5c4622f9a810f035"
2021/01/05 12:13:32 INFO  : 7.txt: Removing failed copy
2021/01/05 12:13:32 DEBUG : 9.txt: MD5 = 12808f4d90a07ec63c617ac251dd80bf (Local file system at /home/aostow/dev/scratch/tmp)
2021/01/05 12:13:32 DEBUG : 9.txt: MD5 = 4d5a6e1a4b99e03d9176cc228250fcc5 (S3 bucket ticfoobar)
2021/01/05 12:13:32 ERROR : 9.txt: corrupted on transfer: MD5 hash differ "12808f4d90a07ec63c617ac251dd80bf" vs "4d5a6e1a4b99e03d9176cc228250fcc5"
2021/01/05 12:13:32 INFO  : 9.txt: Removing failed copy
2021/01/05 12:13:32 DEBUG : 8.txt: MD5 = 5c38c9d77fee15fd82d3ab04b199930b (Local file system at /home/aostow/dev/scratch/tmp)
2021/01/05 12:13:32 DEBUG : 8.txt: MD5 = 688ba85954de6c5d41de1afcc5a47988 (S3 bucket ticfoobar)
2021/01/05 12:13:32 ERROR : 8.txt: corrupted on transfer: MD5 hash differ "5c38c9d77fee15fd82d3ab04b199930b" vs "688ba85954de6c5d41de1afcc5a47988"
2021/01/05 12:13:32 INFO  : 8.txt: Removing failed copy
2021/01/05 12:13:32 INFO  : There was nothing to transfer
2021/01/05 12:13:32 ERROR : Attempt 2/3 failed with 10 errors and: corrupted on transfer: MD5 hash differ "5c38c9d77fee15fd82d3ab04b199930b" vs "688ba85954de6c5d41de1afcc5a47988"
2021/01/05 12:13:32 DEBUG : S3 bucket ticfoobar: Waiting for checks to finish
2021/01/05 12:13:32 DEBUG : S3 bucket ticfoobar: Waiting for transfers to finish
2021/01/05 12:13:32 DEBUG : 1.txt: MD5 = b026324c6904b2a9cb4b88d6d61c81d1 (Local file system at /home/aostow/dev/scratch/tmp)
2021/01/05 12:13:32 DEBUG : 1.txt: MD5 = a5080591791156fe4774811a3dccf0b2 (S3 bucket ticfoobar)
2021/01/05 12:13:32 ERROR : 1.txt: corrupted on transfer: MD5 hash differ "b026324c6904b2a9cb4b88d6d61c81d1" vs "a5080591791156fe4774811a3dccf0b2"
2021/01/05 12:13:32 INFO  : 1.txt: Removing failed copy
2021/01/05 12:13:32 DEBUG : 2.txt: MD5 = f303b7d2f2b87f9e16df05e2bca7c409 (Local file system at /home/aostow/dev/scratch/tmp)
2021/01/05 12:13:32 DEBUG : 2.txt: MD5 = 0b16d17df0f9b7cdd92fdd17c6e4a790 (S3 bucket ticfoobar)
2021/01/05 12:13:32 ERROR : 2.txt: corrupted on transfer: MD5 hash differ "f303b7d2f2b87f9e16df05e2bca7c409" vs "0b16d17df0f9b7cdd92fdd17c6e4a790"
2021/01/05 12:13:32 INFO  : 2.txt: Removing failed copy
2021/01/05 12:13:32 DEBUG : 10.txt: MD5 = d12dd13f4dfe9adef553f0490763d4df (Local file system at /home/aostow/dev/scratch/tmp)
2021/01/05 12:13:32 DEBUG : 10.txt: MD5 = 2fb095006730a0de23037238c82c18a3 (S3 bucket ticfoobar)
2021/01/05 12:13:32 ERROR : 10.txt: corrupted on transfer: MD5 hash differ "d12dd13f4dfe9adef553f0490763d4df" vs "2fb095006730a0de23037238c82c18a3"
2021/01/05 12:13:32 INFO  : 10.txt: Removing failed copy
2021/01/05 12:13:32 DEBUG : 3.txt: MD5 = f2b33fb7b3d0eb95090a16060e6a24f9 (Local file system at /home/aostow/dev/scratch/tmp)
2021/01/05 12:13:32 DEBUG : 3.txt: MD5 = 34d860c92e676503e616b9016cef8327 (S3 bucket ticfoobar)
2021/01/05 12:13:32 ERROR : 3.txt: corrupted on transfer: MD5 hash differ "f2b33fb7b3d0eb95090a16060e6a24f9" vs "34d860c92e676503e616b9016cef8327"
2021/01/05 12:13:32 INFO  : 3.txt: Removing failed copy
2021/01/05 12:13:32 DEBUG : 4.txt: MD5 = fe59a05ba880dc4ad0d50e7401b1648d (Local file system at /home/aostow/dev/scratch/tmp)
2021/01/05 12:13:32 DEBUG : 4.txt: MD5 = dccb21222f3460a5051a7bcb8d2a0211 (S3 bucket ticfoobar)
2021/01/05 12:13:32 ERROR : 4.txt: corrupted on transfer: MD5 hash differ "fe59a05ba880dc4ad0d50e7401b1648d" vs "dccb21222f3460a5051a7bcb8d2a0211"
2021/01/05 12:13:32 INFO  : 4.txt: Removing failed copy
2021/01/05 12:13:32 DEBUG : 7.txt: MD5 = 5d336a7898bfda02ffc516e3b35a7230 (Local file system at /home/aostow/dev/scratch/tmp)
2021/01/05 12:13:32 DEBUG : 7.txt: MD5 = 3eae0a0d8f34840970088605fd72f3eb (S3 bucket ticfoobar)
2021/01/05 12:13:32 ERROR : 7.txt: corrupted on transfer: MD5 hash differ "5d336a7898bfda02ffc516e3b35a7230" vs "3eae0a0d8f34840970088605fd72f3eb"
2021/01/05 12:13:32 INFO  : 7.txt: Removing failed copy
2021/01/05 12:13:32 DEBUG : 5.txt: MD5 = 9f96ef92fe742165873c313662f1f2b8 (Local file system at /home/aostow/dev/scratch/tmp)
2021/01/05 12:13:32 DEBUG : 5.txt: MD5 = 6a4fd2fae2eb3944f81d77671874bf9a (S3 bucket ticfoobar)
2021/01/05 12:13:32 ERROR : 5.txt: corrupted on transfer: MD5 hash differ "9f96ef92fe742165873c313662f1f2b8" vs "6a4fd2fae2eb3944f81d77671874bf9a"
2021/01/05 12:13:32 INFO  : 5.txt: Removing failed copy
2021/01/05 12:13:32 DEBUG : 6.txt: MD5 = 14ba5568c6103ac56facf99277adb053 (Local file system at /home/aostow/dev/scratch/tmp)
2021/01/05 12:13:32 DEBUG : 6.txt: MD5 = 0509f06bf86b3b96497abfc9bc49c793 (S3 bucket ticfoobar)
2021/01/05 12:13:32 ERROR : 6.txt: corrupted on transfer: MD5 hash differ "14ba5568c6103ac56facf99277adb053" vs "0509f06bf86b3b96497abfc9bc49c793"
2021/01/05 12:13:32 INFO  : 6.txt: Removing failed copy
2021/01/05 12:13:32 DEBUG : 8.txt: MD5 = 5c38c9d77fee15fd82d3ab04b199930b (Local file system at /home/aostow/dev/scratch/tmp)
2021/01/05 12:13:32 DEBUG : 8.txt: MD5 = d4c716318427f36a1dc64e746febeb8a (S3 bucket ticfoobar)
2021/01/05 12:13:32 ERROR : 8.txt: corrupted on transfer: MD5 hash differ "5c38c9d77fee15fd82d3ab04b199930b" vs "d4c716318427f36a1dc64e746febeb8a"
2021/01/05 12:13:32 INFO  : 8.txt: Removing failed copy
2021/01/05 12:13:32 DEBUG : 9.txt: MD5 = 12808f4d90a07ec63c617ac251dd80bf (Local file system at /home/aostow/dev/scratch/tmp)
2021/01/05 12:13:32 DEBUG : 9.txt: MD5 = 294b820c00e273d9ad067dd09da2aaef (S3 bucket ticfoobar)
2021/01/05 12:13:32 ERROR : 9.txt: corrupted on transfer: MD5 hash differ "12808f4d90a07ec63c617ac251dd80bf" vs "294b820c00e273d9ad067dd09da2aaef"
2021/01/05 12:13:32 INFO  : 9.txt: Removing failed copy
2021/01/05 12:13:32 INFO  : There was nothing to transfer
2021/01/05 12:13:32 ERROR : Attempt 3/3 failed with 10 errors and: corrupted on transfer: MD5 hash differ "12808f4d90a07ec63c617ac251dd80bf" vs "294b820c00e273d9ad067dd09da2aaef"
2021/01/05 12:13:32 INFO  : 
Transferred:   	       333 / 333 Bytes, 100%, 261 Bytes/s, ETA 0s
Errors:                10 (retrying may help)
Elapsed time:         1.4s

2021/01/05 12:13:32 DEBUG : 10 go routines active
2021/01/05 12:13:32 Failed to copy with 10 errors: last error was: corrupted on transfer: MD5 hash differ "12808f4d90a07ec63c617ac251dd80bf" vs "294b820c00e273d9ad067dd09da2aaef"
2

hello and welcome to the forum,

are you sure, as this is in the config?
server_side_encryption = aws:kms

i had a similar issue, with SSE-C.
i would take a read of this post, with workarounds and details on the beta.
Issues with aws s3 sse-c, getting strange log entries and errors - #8 by ncw

tl;dr
try adding --s3-upload-cutoff 0, this will force rclone to use md5 checksums.

3

Thanks! Using the --s3-upload-cutoff parameter did fix it. Is there any performance impact to using it?

I was a bit confused by the KMS piece of this -- I had thought things were unencrypted because the bucket's property page on the AWS console said default encryption is turned off, but now I see the properties on individual objects report KMS encryption.

4

good, you got it working.

as for performance hit, i have not seen it and even if i did, i have no choice.
any upload without a checksum is like expecting ftp to be reliable.

i also use rclone crypt remotes and never experienced a performance hit.

the weak link is always the internet upload speed, not a fast cpu doing a checksum.

5

This is fixed in the latest beta I think, no need for the workaround with that.

6

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.