Copy file to crypt, is rclone using md5 to compare?

What is the problem you are having with rclone?

based on logs,
--- when copying a local file to a non-crypted remote, rclones does compare the md5 after copy.
--- when copying a local file to a crypted remote, rclone does not seem to compare the md5 after copy.

even when syncing to crypted remote with --checksum, rclone does not seem to be using the md5

What is your rclone version (output from rclone version)

v1.51.0-160-g424554bc-beta

Which OS you are using and how many bits (eg Windows 7, 64 bit)

win10.1909.64

Which cloud storage system are you using? (eg Google Drive)

wasabi

The command you were trying to run (eg rclone copy /tmp remote:tmp)

2020/04/13 10:27:07 DEBUG : rclone: Version "v1.51.0-160-g424554bc-beta" starting with parameters ["C:\data\rclone\scripts\rr\other\test\rclone-v1.51.0-160.exe" "sync" "C:\path\to\local\folder" "secret1:\folder" "--log-level=DEBUG" "--log-file=C:\data\rclone\scripts\rr\other\test\log.test.check.txt" "--checksum"]
2020/04/13 10:27:07 DEBUG : Using RCLONE_CONFIG_PASS password.

A log from the command with the -vv flag (eg output from rclone -vv copy /tmp remote:tmp)

here is the log for a file copied to a non-crypted remote

2020/04/13 10:25:48 DEBUG : rclone: Version "v1.51.0-160-g424554bc-beta" starting with parameters ["C:\data\rclone\scripts\rr\other\test\rclone-v1.51.0-160.exe" "sync" "C:\path\to\local\folder" "remote:\folder" "--log-level=DEBUG" "--log-file=C:\data\rclone\scripts\rr\other\test\log.test.check.txt"]
2020/04/13 10:25:48 DEBUG : Using RCLONE_CONFIG_PASS password.
2020/04/13 10:25:48 DEBUG : Using config file from "c:\data\rclone\scripts\rclone.conf"
2020/04/13 10:25:49 DEBUG : S3 bucket aliasremote path folder: Waiting for checks to finish
2020/04/13 10:25:49 DEBUG : S3 bucket aliasremote path folder: Waiting for transfers to finish
2020/04/13 10:25:49 DEBUG : d/d.txt: MD5 = 6a59d1ee75e8c6d7ad8b74b7759799b8 OK
2020/04/13 10:25:49 INFO : d/d.txt: Copied (new)

here is the log for a file copied to a crypted remote, no MD5
2020/04/13 10:27:07 DEBUG : rclone: Version "v1.51.0-160-g424554bc-beta" starting with parameters ["C:\data\rclone\scripts\rr\other\test\rclone-v1.51.0-160.exe" "sync" "C:\path\to\local\folder" "secret1:\folder" "--log-level=DEBUG" "--log-file=C:\data\rclone\scripts\rr\other\test\log.test.check.txt" "--checksum"]
2020/04/13 10:27:07 DEBUG : Using RCLONE_CONFIG_PASS password.
2020/04/13 10:27:07 DEBUG : Using config file from "c:\data\rclone\scripts\rclone.conf"
2020/04/13 10:27:07 DEBUG : Encrypted drive 'secret1:/folder': Waiting for checks to finish
2020/04/13 10:27:07 DEBUG : Encrypted drive 'secret1:/folder': Waiting for transfers to finish
2020/04/13 10:27:07 INFO : d/d.txt: Copied (new)

With the file encrypted, the MD5 would be different from the source so you can't compare checksums on an encrypted remote.

That's why you have to use cryptcheck over check.

Each time you upload, the md5sum changes due to the crypt:

felix@gemini:~$ rclone lsf --hash MD5 --format hp --separator "  " --files-only GD:crypt/rs88l6p51j4kp0g9if3j294ts0
fdfd9451f01b94cd2fe302b9ab938808  rs88l6p51j4kp0g9if3j294ts0
felix@gemini:~$ rclone delete gcrypt:hosts
rclone felix@gemini:~$ rclone copy /etc/hosts gcrypt:
felix@gemini:~$ rclone lsf --hash MD5 --format hp --separator "  " --files-only GD:crypt/rs88l6p51j4kp0g9if3j294ts0
458f8383693866e450706aeb00a0ea3a  rs88l6p51j4kp0g9if3j294ts0

and that never matches the source.

felix@gemini:~$ md5sum /etc/hosts
c0c69fcfeb6162dd8065b5a5a61e70e4  /etc/hosts

thanks but, perhaps i am confused.

i understand that the md5 of the local file will not match the hash of the crypted file.

but when rclone is copying, rclone is saving the md5 of the crypted file, as metadata.
copy.

Compare what?

The source md5sum would never match the destination md5sum since it changes each time because of encryption.

i understand what you are writing.

am i correct that,
when rclone uploads a file to a crypt, it saves the checksum of the decrypted file as metadata?

No, rclone doesn't store anything as that's all on the provider.

Based on the provider remote, metadata is uploaded to that provider.

So in my example, Google stores the MD5SUMs as part of the metadata of a file.

There is no metadata kept anywhere in rclone (assuming we are excluding the cache backend for this discussion).

ok. thanks

is there a reason why rclone cannot save the md5 of the local file, as metadata?

There have been a few issues of keeping cached metadata for things so there isn't a real reason other than time and effort to implement.

I would think most of the time, you'd recheck the md5sum to make sure the file didn't change anyway so not sure the use case of keeping it.

I'm sure there are flows out there, but I would not see a use for myself since I encrypt all my data anyway.

1 Like

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.