Copyto AWS S3 with --checksum posts error "checksum is in use but the source and destination have no hashes in common; failing back to size-only"

Voyager · December 26, 2024, 3:56pm

What is the problem you are having with rclone?

I'm copytoing a file from my local machine to AWS S3. When I do, I get an error:

2024/12/26 09:47:39 NOTICE: Encrypted drive 'cryptbucket:/': --checksum is in use but the source and destination have no hashes in common; falling back to --size-only

The file in question is the output of ls -lRa and because the output is likely to change with a few characters replaced, size is not the best method to compare the file.

A google search over the forum yielded two results that don't seem to fit my specific use case. Without the "s3" query parameter, it brought up others but seemed to deal with Google Drive.

When I inspect the S3 object in the AWS console, the only metadata it has "x-amz-meta-mtime". When I check other files uploaded to S3 (albeit with the copy command), I see "x-amz-meta-md5chksum" in addition to "...mtime".

I have never enabled --s3-disable-checksum so I'm not sure why the file would not have the checksum metadata.

As a troubleshooting step, I deleted the file and then re-uploaded it and then tried to copy it again. Please see full logs at the very bottom.

Run the command 'rclone version' and share the full output of the command.

$ rclone --version
rclone v1.68.2
- os/version: linuxmint 21.3 (64 bit)
- os/kernel: 6.8.0-50-generic (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.23.3
- go/linking: static
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

AWS S3

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

rclone copyto -vv --checksum --s3-storage-class "STANDARD" --s3-no-check-bucket \
  /path/to/ls-lRa.txt cipher:/ls-lRa.txt

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

[aws-s3]
type = s3
provider = AWS
env_auth = true
access_key_id = XXX
secret_access_key = XXX
region = us-east-1
server_side_encryption = AES256

[cryptbucket]
type = crypt
remote = aws-s3:<<REDACTED>>/archives/<<HOSTNAME>>/<<PERSONAL_DIRECTORY>>
password = XXX

A log from the command that you were trying to run with the `-vv` flag

$ rclone delete -vv cryptbucket:/ls-lRa.txt
2024/12/26 10:41:18 DEBUG : rclone: Version "v1.68.2" starting with parameters ["rclone" "delete" "-vv" "cryptbucket:/ls-lRa.txt"]
2024/12/26 10:41:18 DEBUG : Creating backend with remote "cryptbucket:/ls-lRa.txt"
2024/12/26 10:41:18 DEBUG : Using config file from "/home/user/.config/rclone/rclone.conf"
2024/12/26 10:41:18 DEBUG : Creating backend with remote "aws-s3:<<REDACTED>>/archives/<<HOSTNAME>>/<<PERSONAL_DIRECTORY>>/81rtsqbp5i1tdhqs8dqdon4a4k7d6508pvdd2k6vdc4o0omsffs0"
2024/12/26 10:41:18 DEBUG : fs cache: adding new entry for parent of "aws-s3:<<REDACTED>>/archives/<<HOSTNAME>>/<<PERSONAL_DIRECTORY>>/81rtsqbp5i1tdhqs8dqdon4a4k7d6508pvdd2k6vdc4o0omsffs0", "aws-s3:<<REDACTED>>/archives/<<HOSTNAME>>/<<PERSONAL_DIRECTORY>>"
2024/12/26 10:41:18 DEBUG : fs cache: adding new entry for parent of "cryptbucket:/ls-lRa.txt", "cryptbucket:"
2024/12/26 10:41:18 DEBUG : Waiting for deletions to finish
2024/12/26 10:41:18 DEBUG : media: Excluded
2024/12/26 10:41:18 INFO  : ls-lRa.txt: Deleted
2024/12/26 10:41:18 DEBUG : 6 go routines active


$ rclone copyto -vv --checksum --s3-storage-class "STANDARD" --s3-no-check-bucket /path/to/dir/ls-lRa.txt cryptbucket:/ls-lRa.txt
2024/12/26 10:41:23 DEBUG : rclone: Version "v1.68.2" starting with parameters ["rclone" "copyto" "-vv" "--checksum" "--s3-storage-class" "STANDARD" "--s3-no-check-bucket" "/path/to/dir/ls-lRa.txt" "cryptbucket:/ls-lRa.txt"]
2024/12/26 10:41:23 DEBUG : Creating backend with remote "/path/to/dir/ls-lRa.txt"
2024/12/26 10:41:23 DEBUG : Using config file from "/home/user/.config/rclone/rclone.conf"
2024/12/26 10:41:23 DEBUG : fs cache: adding new entry for parent of "/path/to/dir/ls-lRa.txt", "/path/to/dir"
2024/12/26 10:41:23 DEBUG : Creating backend with remote "cryptbucket:/"
2024/12/26 10:41:23 DEBUG : Creating backend with remote "aws-s3:<<REDACTED>>/archives/<<HOSTNAME>>/<<PERSONAL_DIRECTORY>>"
2024/12/26 10:41:23 DEBUG : aws-s3: detected overridden config - adding "{mH4YA}" suffix to name
2024/12/26 10:41:23 DEBUG : fs cache: renaming cache item "aws-s3:<<REDACTED>>/archives/<<HOSTNAME>>/<<PERSONAL_DIRECTORY>>" to be canonical "aws-s3{mH4YA}:<<REDACTED>>/archives/<<HOSTNAME>>/<<PERSONAL_DIRECTORY>>"
2024/12/26 10:41:23 DEBUG : fs cache: switching user supplied name "aws-s3:<<REDACTED>>/archives/<<HOSTNAME>>/<<PERSONAL_DIRECTORY>>" for canonical name "aws-s3{mH4YA}:<<REDACTED>>/archives/<<HOSTNAME>>/<<PERSONAL_DIRECTORY>>"
2024/12/26 10:41:23 DEBUG : ls-lRa.txt: Need to transfer - File not found at Destination
2024/12/26 10:41:23 DEBUG : ls-lRa.txt: Computing md5 hash of encrypted source
2024/12/26 10:41:23 DEBUG : ls-lRa.txt: md5 = 0265f21d47bebd360f44642609ea035e OK
2024/12/26 10:41:23 INFO  : ls-lRa.txt: Copied (new)
2024/12/26 10:41:23 INFO  :
Transferred:      143.426 KiB / 143.426 KiB, 100%, 0 B/s, ETA -
Transferred:            1 / 1, 100%
Elapsed time:         0.3s

2024/12/26 10:41:23 DEBUG : 7 go routines active


$ rclone copyto -vv --checksum --s3-storage-class "STANDARD" --s3-no-check-bucket /path/to/dir/ls-lRa.txt cryptbucket:/ls-lRa.txt
2024/12/26 10:41:30 DEBUG : rclone: Version "v1.68.2" starting with parameters ["rclone" "copyto" "-vv" "--checksum" "--s3-storage-class" "STANDARD" "--s3-no-check-bucket" "/path/to/dir/ls-lRa.txt" "cryptbucket:/ls-lRa.txt"]
2024/12/26 10:41:30 DEBUG : Creating backend with remote "/path/to/dir/ls-lRa.txt"
2024/12/26 10:41:30 DEBUG : Using config file from "/home/user/.config/rclone/rclone.conf"
2024/12/26 10:41:30 DEBUG : fs cache: adding new entry for parent of "/path/to/dir/ls-lRa.txt", "/path/to/dir"
2024/12/26 10:41:30 DEBUG : Creating backend with remote "cryptbucket:/"
2024/12/26 10:41:30 DEBUG : Creating backend with remote "aws-s3:<<REDACTED>>/archives/<<HOSTNAME>>/<<PERSONAL_DIRECTORY>>"
2024/12/26 10:41:30 DEBUG : aws-s3: detected overridden config - adding "{mH4YA}" suffix to name
2024/12/26 10:41:30 DEBUG : fs cache: renaming cache item "aws-s3:<<REDACTED>>/archives/<<HOSTNAME>>/<<PERSONAL_DIRECTORY>>" to be canonical "aws-s3{mH4YA}:<<REDACTED>>/archives/<<HOSTNAME>>/<<PERSONAL_DIRECTORY>>"
2024/12/26 10:41:30 DEBUG : fs cache: switching user supplied name "aws-s3:<<REDACTED>>/archives/<<HOSTNAME>>/<<PERSONAL_DIRECTORY>>" for canonical name "aws-s3{mH4YA}:<<REDACTED>>/archives/<<HOSTNAME>>/<<PERSONAL_DIRECTORY>>"
2024/12/26 10:41:30 NOTICE: Encrypted drive 'cryptbucket:/': --checksum is in use but the source and destination have no hashes in common; falling back to --size-only
2024/12/26 10:41:30 DEBUG : ls-lRa.txt: Size of src and dst objects identical
2024/12/26 10:41:30 DEBUG : ls-lRa.txt: Unchanged skipping
2024/12/26 10:41:30 INFO  :
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:         0.1s

2024/12/26 10:41:30 DEBUG : 6 go routines active

Please note: I used find and replace to redact sensitive or personal info to ensure consistency but I may have missed something. I'm happy to provide more information if needed, with the exception of the s3 bucket name since publishing this can make my bill explode.

PS: I also know it's the holidays. Please prioritize yourselves and your family. I'll keep plugging away at this and if I have an update, I'll edit accordingly.

asdffdsa · December 26, 2024, 4:00pm

welcome to the forum,

Hashes are not stored for crypt

Voyager · December 26, 2024, 4:07pm

I think it does because other files I've uploaded for crypt have the "x-amz-meta-md5chksum" metadata field.

In fact, I just did a copy of the directory with the ls-lRa file in it along with a backup tar file and the ls-lRa.txt file does NOT have the metadata field whereas the tar file does.

Maybe crypt doesn't do it for text files?

asdffdsa · December 26, 2024, 5:30pm

i think that is expected behaviour, tho, hard to know without seeing the debug log for the tar file upload.

i will assume the following. let me know if i am correct or not?
the tar file file size is larger than 5MiB??

afiak, in your case, cannot compare the two hashes. each file will result in a different hash.

source file is non-crypted
the dest file is crypted

Voyager · December 26, 2024, 6:51pm

Sorry for my short reply first time around. Thank you for the welcome! Glad to be here!

Here are the logs for the copy command:

$ rclone copy -vv --checksum --s3-storage-class "STANDARD" --s3-no-check-bucket /path/to/ cryptbucket:/
2024/12/26 11:03:07 DEBUG : rclone: Version "v1.68.2" starting with parameters ["rclone" "copy" "-vv" "--checksum" "--s3-storage-class" "STANDARD" "--s3-no-check-bucket" "/path/to/" "cryptbucket:/"]
2024/12/26 11:03:07 DEBUG : Creating backend with remote "/path/to/"
2024/12/26 11:03:07 DEBUG : Using config file from "/home/user/.config/rclone/rclone.conf"
2024/12/26 11:03:07 DEBUG : fs cache: renaming cache item "/path/to/" to be canonical "/path/to"
2024/12/26 11:03:07 DEBUG : Creating backend with remote "cryptbucket:/"
2024/12/26 11:03:07 DEBUG : Creating backend with remote "aws-s3:<<REDACTED>>/archives/<<HOSTNAME>>/<<PERSONAL_DIRECTORY>>"
2024/12/26 11:03:07 DEBUG : aws-s3: detected overridden config - adding "{mH4YA}" suffix to name
2024/12/26 11:03:08 DEBUG : fs cache: renaming cache item "aws-s3:<<REDACTED>>/archives/<<HOSTNAME>>/<<PERSONAL_DIRECTORY>>" to be canonical "aws-s3{mH4YA}:<<REDACTED>>/archives/<<HOSTNAME>>/<<PERSONAL_DIRECTORY>>"
2024/12/26 11:03:08 DEBUG : fs cache: switching user supplied name "aws-s3:<<REDACTED>>/archives/<<HOSTNAME>>/<<PERSONAL_DIRECTORY>>" for canonical name "aws-s3{mH4YA}:<<REDACTED>>/archives/<<HOSTNAME>>/<<PERSONAL_DIRECTORY>>"
2024/12/26 11:03:08 DEBUG : ls-lRa.txt: Need to transfer - File not found at Destination
2024/12/26 11:03:08 DEBUG : backup-config.tar.bz: Need to transfer - File not found at Destination
2024/12/26 11:03:08 DEBUG : Encrypted drive 'cryptbucket:/': Waiting for checks to finish
2024/12/26 11:03:08 DEBUG : Encrypted drive 'cryptbucket:/': Waiting for transfers to finish
2024/12/26 11:03:08 DEBUG : ls-lRa.txt: Computing md5 hash of encrypted source
2024/12/26 11:03:08 DEBUG : backup-config.tar.bz: Computing md5 hash of encrypted source
2024/12/26 11:03:08 DEBUG : ls-lRa.txt: md5 = 7bdb1e1409e072929bbbd4a5c68e6c09 OK
2024/12/26 11:03:08 INFO  : ls-lRa.txt: Copied (new)
2024/12/26 11:03:08 DEBUG : pglho14ll2fvo07079si03moer253tqtk7ubik9k0lsldctjhvtg: open chunk writer: started multipart upload: UWOVSCysGr6CS2aqpe3Rv6LxC4FI6SnisokIsctwyCFy52v0rM0o01bBD9U_Cw9PQjTSXV05FqeW8udV.W6_IdSfwRN9b20HasdWqJUG4EQV_17tnCapTKnpMX7w_7nb
2024/12/26 11:03:08 DEBUG : backup-config.tar.bz: multipart upload: starting chunk 0 size 5Mi offset 0/327.150Mi
... -- CHUNKING --
2024/12/26 11:03:20 DEBUG : pglho14ll2fvo07079si03moer253tqtk7ubik9k0lsldctjhvtg: multipart upload "UWOVSCysGr6CS2aqpe3Rv6LxC4FI6SnisokIsctwyCFy52v0rM0o01bBD9U_Cw9PQjTSXV05FqeW8udV.W6_IdSfwRN9b20HasdWqJUG4EQV_17tnCapTKnpMX7w_7nb" finished
2024/12/26 11:03:20 DEBUG : pglho14ll2fvo07079si03moer253tqtk7ubik9k0lsldctjhvtg: Multipart upload Etag: 65ecce323b5494996fda8b132aeef731-66 OK
2024/12/26 11:03:20 DEBUG : backup-config.tar.bz: md5 = b5e41ebab34b651c376432106b2510fd OK
2024/12/26 11:03:20 INFO  : backup-config.tar.bz: Copied (new)
2024/12/26 11:03:20 INFO  :
Transferred:      327.290 MiB / 327.290 MiB, 100%, 27.274 MiB/s, ETA 0s
Transferred:            2 / 2, 100%
Elapsed time:        12.6s

2024/12/26 11:03:20 DEBUG : 14 go routines active

Yes it is! The backup file is 328mb

I would have expected that the hash the hash stored in "x-amz-meta-md5chksum" be the hash prior to being encrypted so that it can be compared for future runs of the command. If not that, then hashing the localfile to compare it against the hashed value of the binary file.

As I'm investigating, I found something interesting:

I took the value in the hash and and decoded it per the instructions on the s3 page:

$ echo 'teQeurNLZRw3ZDIQayUQ/Q==' | base64 -d | hexdump
0000000 e4b5 ba1e 4bb3 1c65 6437 1032 256b fd10
0000010

In the logs above, you can see that the md5sum of the backup is (I added spaces for visibility):
b5e4 1eba b34b 651c 3764 3210 6b25 10fd.

It looks like two sets of characters are swapped for each set of characters! I've never seen this before.

Oddly enough, when I do copyto specifically for the backup file, I get the same error "--checksum is in use but the source and destination have no hashes in common; falling back to --size-only".

I'm wondering if the crypt documentation would need to be updated to indicate what the value of "x-amz-meta-md5chksum" is.

That said, returning to my original question: what would be the best way to test whether the text file has changed and then uploading an updated file?

asdffdsa · December 26, 2024, 8:09pm

good, i was expecting that.
for uploads that are single part, below 5MiB, the ETag is the md5 of the file, so no need to add x-amz-meta-md5chksum
for uploads that are multiple part, above 5MiB, the ETag is not the md5 of the file, so need to add x-amz-meta-md5chksum
if you want a deep dive into that, check out this topic, were i helped get rclone to use ETags.
https://forum.rclone.org/t/in-some-cases-rclone-does-not-use-etag-to-verify-files/36095

well, if the hash of the unencrypted file was saved as metadata, that would be a leak of private information.
imo, not something you want to a crypted remote.

fwiw, that is actually the sort of bug i found at wasabi.
when using sse-c encryption, wasabi leaks the hash of the non-crypted file

i contacted wasabi, years ago about it, but to this day, they have not fixed it.
hear back from wasabi

it does not matter if the file is text or binary. the same rules apply.

hash
modtime
size

if you want to use hash with crypt, might try rclone check --download

Voyager · December 26, 2024, 8:39pm

Thanks for all your help @asdffdsa

I was able to figure out a decent solution but I can't speak to how many calls this uses (and a bit lazy to inspect cloudtrail logs )

You can use rclone cryptcheck with the --error option to create a file and then feed that file into rclone copy --files-from:

echo "rclone: verifying checksum between local files and crypt"
CRYPTCHECK_FILE=$(mktemp /tmp/rclone_cryptcheck_file.XXXXXX.txt)
rclone cryptcheck -vv --one-way --differ $CRYPTCHECK_FILE \
    --missing-on-dst $CRYPTCHECK_FILE /path/to cryptbucket:/

# If the file is of non-zero size, then do the copy
if [ -s $CRYPTCHECK_FILE ]; then 
  echo "cryptcheck returned a list of diffs"
  echo "rclone: copying backup fies to AWS S3"
  rclone copy -vv --s3-storage-class "STANDARD" --s3-no-check-bucket \
    --update --use-server-modtime --files-from $CRYPTCHECK_FILE \
    /path/to cryptbucket:/
fi

You can do this one file at a time by adding the filename to the cryptcheck command on the src side:

rclone cryptcheck -vv --one-way --differ $CRYPTCHECK_FILE --missing-on-dst $CRYPTCHECK_FILE /path/to/ls-lRa.txt cryptbucket:/

This will filter cryptbucket:/ for the specific file.

Voyager · December 26, 2024, 8:42pm

It could go the other way: encrypt the localfile and hash that and then compare it against the md5sum on crypt.

Maybe this could be a feature added in a future release?

system · December 29, 2024, 8:42pm

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.