Does a "rclone check" on similar remotes still compute hashes?

What is the problem you are having with rclone?

when doing a "rclone check" between two remotes that are encrypted, even though rclone is not decrypting the files, the hash check should accurately compare the files, not? I mean, if there are two files that are encrypted with the same encryption to different remotes, their hash should still match, no? The reason i ask is, when doing a -vv log, only one remote lists the algorith used for the checksum. is that normal?

i guess my actual question is, when doing a check between two encrypted remotes, do i need to decrypt the data using cryptcheck, or will i be fine using the regular check, which i assume will require fewer resources?

What is your rclone version (output from rclone version)

1.55.1

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Windows Hyper-V Server 2019

Which cloud storage system are you using? (eg Google Drive)

Microsoft OneDrive for Business

The command you were trying to run (eg rclone copy /tmp remote:tmp)

C:\rclone 1.55.1>rclone check --verbose -vv "OneDriveWy:TBA Server\Human Resource" "OneDriveGina:Server 5-2-21\Human Resource"

A log from the command with the -vv flag

C:\rclone 1.55.1>rclone check --verbose -vv "OneDriveWy:TBA Server\Human Resource" "OneDriveGina:Server 5-2-21\Human Resource"
2021/05/17 13:19:44 DEBUG : Using config file from "C:\\Users\\Administrator\\.config\\rclone\\rclone.conf"
2021/05/17 13:19:44 DEBUG : rclone: Version "v1.55.1" starting with parameters ["rclone" "check" "--verbose" "-vv" "OneDriveWy:TBA Server\\Human Resource" "OneDriveGina:Server 5-2-21\\Human Resource"]
2021/05/17 13:19:44 DEBUG : Creating backend with remote "OneDriveWy:TBA Server\\Human Resource"
2021/05/17 13:19:45 DEBUG : fs cache: renaming cache item "OneDriveWy:TBA Server\\Human Resource" to be canonical "OneDriveWy:TBA Server/Human Resource"
2021/05/17 13:19:45 DEBUG : Creating backend with remote "OneDriveGina:Server 5-2-21\\Human Resource"
2021/05/17 13:19:46 DEBUG : fs cache: renaming cache item "OneDriveGina:Server 5-2-21\\Human Resource" to be canonical "OneDriveGina:Server 5-2-21/Human Resource"
2021/05/17 13:19:46 DEBUG : One drive root 'Server 5-2-21/Human Resource': Waiting for checks to finish
2021/05/17 13:19:47 DEBUG : at58g15q19p1e7d9r5t36gd8maur8ourfopjft3100oiujnh633ce12rjfci66nqcai76kk4do976: QuickXorHash = a---6 OK
2021/05/17 13:19:47 DEBUG : guvkdsq842eudce0e28emmcqq2eklvodbdoum2luj4f0vtl1bcrq9eqlo2vn7uc0hp0j21rhikcvm: QuickXorHash = 9---c OK
2021/05/17 13:19:47 DEBUG : edcsfnor4vfqmvdk0m69kg76rnf42gjc9p4v5dgulaaf3tb06mrg: QuickXorHash = 4---c OK
2021/05/17 13:19:47 DEBUG : edcsfnor4vfqmvdk0m69kg76rnf42gjc9p4v5dgulaaf3tb06mrg: OK
2021/05/17 13:19:47 DEBUG : pm9dg5u4ag3d0p0v9pcf672ujro5jqcfesn3ejqud20v4o6s8mu9bjg8nbr466m0m63ta36outibg: QuickXorHash = 7---0 OK
2021/05/17 13:19:47 DEBUG : at58g15q19p1e7d9r5t36gd8maur8ourfopjft3100oiujnh633ce12rjfci66nqcai76kk4do976: OK
2021/05/17 13:19:47 DEBUG : guvkdsq842eudce0e28emmcqq2eklvodbdoum2luj4f0vtl1bcrq9eqlo2vn7uc0hp0j21rhikcvm: OK
2021/05/17 13:19:47 DEBUG : pm9dg5u4ag3d0p0v9pcf672ujro5jqcfesn3ejqud20v4o6s8mu9bjg8nbr466m0m63ta36outibg: OK

Each time you'd upload a file, the hash would change so you can't compare hashes on encrypted remotes.

You'd want to check out:

https://rclone.org/commands/rclone_cryptcheck/

if i upload the same file to 2 different crypt remotes, and they have the same encryption options, the file name on both remotes is identical. therefore, the encrypted file contents should be identical too, no?

The contents are identical.

The hash on the file is not.

is that the nature of encryption....each time you run a file through an encryption algorithm, the output will arrive at something different?

Yes, that's how encryption works.

Here is an example:

felix@gemini:~$ rclone lsf GD:crypt
95tj3q4gj5ban13ppu0kisguco
bmt5prcdids396lgi777nm2dj8/
rs88l6p51j4kp0g9if3j294ts0
smu5ej34ujbdoip1cm3mlk92q4/
tnvepu36qiohcun8v84ddhsam0/
vib85lf1f6rrl3395nce8noc10/
felix@gemini:~$ rclone md5sum GD:crypt/rs88l6p51j4kp0g9if3j294ts0
67a3bc7d3600b1cd393b54f527d3b4d7  rs88l6p51j4kp0g9if3j294ts0
felix@gemini:~$ md5sum /etc/hosts
38b847bf753f092fa1fac6e5e4018155  /etc/hosts
felix@gemini:~$ rclone delete gcrypt:hosts
felix@gemini:~$ rclone copy /etc/hosts gcrypt:
felix@gemini:~$ rclone lsf GD:crypt
95tj3q4gj5ban13ppu0kisguco
bmt5prcdids396lgi777nm2dj8/
rs88l6p51j4kp0g9if3j294ts0
smu5ej34ujbdoip1cm3mlk92q4/
tnvepu36qiohcun8v84ddhsam0/
vib85lf1f6rrl3395nce8noc10/
felix@gemini:~$ rclone md5sum GD:crypt/rs88l6p51j4kp0g9if3j294ts0
72199cdbeb43d5d5913f11544e864df6  rs88l6p51j4kp0g9if3j294ts0

I uploaded, got a new hash on the file, deleted, uploaded it again and got a new hash on the file.

so why does the file name compute to the exact same thing every time?

It's documented on the crypt remote page here:

https://rclone.org/crypt/#example

this is literally telling me i can use rclone check between two different encrypted remotes, as long as they have the same password, and that i don't need to use cryptcheck. so therefore, the hashes on the files in the different remotes should be identical:

For example, let's say you have your original remote at remote: with the encrypted version at eremote: with path remote:crypt. You would then set up the new remote remote2: and then the encrypted version eremote2: with path remote2:crypt using the same passwords as eremote:.

To sync the two remotes you would do

rclone sync -i remote:crypt remote2:crypt

And to check the integrity you would do

rclone check remote:crypt remote2:crypt

Note how it doesn't check the hashes.

felix@gemini:/opt/rclone$ rclone copy /etc/hosts test1:
felix@gemini:/opt/rclone$ rclone sync test1: test2:
felix@gemini:/opt/rclone$ rclone check test1: test2: -vv
2021/05/17 17:41:10 DEBUG : Using config file from "/opt/rclone/rclone.conf"
2021/05/17 17:41:10 DEBUG : rclone: Version "v1.55.1" starting with parameters ["rclone" "check" "test1:" "test2:" "-vv"]
2021/05/17 17:41:10 DEBUG : Creating backend with remote "test1:"
2021/05/17 17:41:11 DEBUG : Creating backend with remote "GD:test1"
2021/05/17 17:41:11 DEBUG : Creating backend with remote "test2:"
2021/05/17 17:41:11 DEBUG : Creating backend with remote "GD:test2"
2021/05/17 17:41:11 DEBUG : Encrypted drive 'test2:': Waiting for checks to finish
2021/05/17 17:41:11 DEBUG : hosts: OK - could not check hash
2021/05/17 17:41:11 NOTICE: Encrypted drive 'test2:': 0 differences found
2021/05/17 17:41:11 NOTICE: Encrypted drive 'test2:': 1 hashes could not be checked
2021/05/17 17:41:11 NOTICE: Encrypted drive 'test2:': 1 matching files
2021/05/17 17:41:11 INFO  :
Transferred:   	         0 / 0 Bytes, -, 0 Bytes/s, ETA -
Checks:                 1 / 1, 100%
Elapsed time:         0.6s

2021/05/17 17:41:11 DEBUG : 7 go routines active
felix@gemini:/opt/rclone$

And with cryptcheck.

felix@gemini:/opt/rclone$ rclone cryptcheck test1: test2:
2021/05/17 17:41:58 NOTICE: Encrypted drive 'test2:': 0 differences found
2021/05/17 17:41:58 NOTICE: Encrypted drive 'test2:': 1 matching files

Here is the page on cryptcheck:

https://rclone.org/commands/rclone_cryptcheck/

rclone cryptcheck checks a remote against a crypted remote. This is the equivalent of running rclone check, but able to check the checksums of the crypted remote.

ok, this is exactly what i was running into. so what DOES it check if i'm using rclone check? just the file size and modification time?

and also, why does my log not show "could not check hash" like yours does? :confused:

What does your rclone.conf look like with the passwords/keys/etc/removed?

Same would be size/date/time as you stated.

[OneDriveWy]
type = onedrive
drive_id = 
drive_type = business

[OneDriveWyEncrypted]
type = crypt
remote = OneDriveWy:TBA Server
filename_encryption = standard
directory_name_encryption = false

[OneDriveGina]
type = onedrive
token =
drive_type = business

are the remotes similar that you're using? perhaps that's why it's not checking the hashes. as you can see on my log in the initial post it does compute a QuickXorHash, and matches it to the other same file, for which it just says "OK"; i'm using OneDriveBusiness to OneDriveBusiness.

but i also note that it never explicitly says "Using QuickXorHash for hash comparison" like it does on the cryptcheck. i guess now i'm just curious why it's mentioning the QuickXorHash in the debug log.

and it also begs the question, if i'm backing up an encrypted remote, and i'm not encrypting/decrypting the data, what guarantee of safe transfer do i have, if it doesn't do a hash checksum as it transferring the files?

If you are making a backup of a crypt by copying the underlying encrypted data, then rclone check will work just fine and it will check the checksums. This appears to be what you are doing.

If you copy the same file to two crypt remotes then the hashes will be different as the file is different each time it is encrypted, even if the keys, etc are the same.

Let's say you have two underlying remotes called A and B and the crypt versions cryptA and cryptB.

If your backup does rclone sync A: B: then rclone check A: B: will be an excellent check, checking the checksums.

If your backup does rclone sync cryptA: cryptB then rclone check cryptA: cryptB: will not check checksums. You could use rclone cryptcheck in this case, but note that it will involve downloading the data from one side and since in your case A and B are remote then is undesirable.

Looking through the thread I think you are doing rclone sync A: B: and your rclone check A: B: seems to be checking hashes just fine.

matches exactly what i was seeing.

if i may humbly make a suggestion, when doing an "rclone check", having the little "Using QuickXorHash for Hash Comparisons" notice like we get when doing an "rclone cryptcheck" would be nice, because at the moment, the only way to get the hash comparison info is to extract the debug log.

Thank you for the clarification.

Jared

Good idea - I push a little patch into the latest beta which does exactly that - it will be released in 1.56 :slight_smile:

1 Like

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.