Running copy command after cryptcheck failed isn't fixing the hash error from cryptcheck

Help and Support
1

Hi Everyone,

I'm running v1.57.0. I am backing up my local files to OneDrive using an encrypted remote. The problem I'm having is that after running copy, when I run cryptcheck it gives me an error on one of the files. The error is that hashes differ between the local file and the file in OD. So I ran copy again but that isn't fixing the cryptcheck error. Is there another way to fix it? I assumed the copy command would rewrite a file if there was an update to it or hashes didnt match but it doesn't seem to be working as I thought it would. What am I missing?

Thank you!

2

hello,

the answers to the questions in the template.
when you posted there was a template of questions, none of which were answered.
we cannot see into your computer, so help us to help you, answer all the questions.

3

Here you go.

What is the problem you are having with rclone?

I am backing up my local files to OneDrive using an encrypted remote. The problem I'm having is that after running copy, when I run cryptcheck it gives me an error on one of the files. The error is that hashes differ between the local file and the file in OD. So I ran copy again but that isn't fixing the cryptcheck error. Is there another way to fix it? I assumed the copy command would rewrite a file if there was an update to it or hashes didnt match but it doesn't seem to be working as I thought it would.

Run the command 'rclone version' and share the full output of the command.

rclone v1.57.0

  • os/version: Microsoft Windows 10 Pro 2009 (64 bit)
  • os/kernel: 10.0.19042.1415 (x86_64)
  • os/type: windows
  • os/arch: amd64
  • go/version: go1.17.2
  • go/linking: dynamic
  • go/tags: cmount

Are you on the latest version of rclone? You can validate by checking the version listed here: Rclone downloads
-->
Yes

Which cloud storage system are you using? (eg Google Drive)

OneDrive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copy /path/to/files encryptedremote:path -P --transfers 8
rclone cryptcheck /path/to/files encryptedremote:path
rclone copy /path/to/files encryptedremote:path -P --transfers 8
4

thanks but please understand that we cannot see into your computer screen.

need to see a debug log, with the exact command, the exact error,
including at least the top 20 lines of debug log.
run the cryptcheck command as that giving the error.

5

I'll try to get the log output later today when I can rerun the command but the error was that the hash of the file didn't match between my local copy and the backup copy.

But I am hoping it is possible to discuss whether what I am doing should theoretically fix the issue without the logs. My understanding is that copy command should detect that there is a difference between the file on my local drive and the encrypted on OneDrive, and therefore should rewrite it. Is this understanding correct or not?

6

perhaps just delete the file in onedrive and copy again.

if still problem, then run cryptcheck that single file and post the entire debug log.

7

as i understand it,
rclone copy to a crypt compares modtime and size, not hash.

so it is possible for rclone to copy the file, notice that the modtime and size match and not recopy
but when running cryptcheck, it notices the hash mismatch.

8

Should / Would are all great but without seeing the data as to what's happening, it's a guessing game to what the problem is.

If you copy a file from A->B and B isn't the same as A, it copies/overwrites the file as that's how copy works.

Maybe you found a bug? Maybe you used a wrong flag? Maybe you have a version from years ago? Maybe you have a typo in a flag / file / etc?

9

hi, i posted just before your last post.

do you think what i wrote is plausible?

10

If you have a crypted remote and you are using cryptcheck, it should match (assuming the files are the same)

felix@gemini:~/test$ rclone copy /home/felix/test gcrypt: -vvv
2022/01/26 16:22:33 DEBUG : Setting --config "/opt/rclone/rclone.conf" from environment variable RCLONE_CONFIG="/opt/rclone/rclone.conf"
2022/01/26 16:22:33 DEBUG : rclone: Version "v1.57.0" starting with parameters ["rclone" "copy" "/home/felix/test" "gcrypt:" "-vvv"]
2022/01/26 16:22:33 DEBUG : Creating backend with remote "/home/felix/test"
2022/01/26 16:22:33 DEBUG : Using config file from "/opt/rclone/rclone.conf"
2022/01/26 16:22:33 DEBUG : Creating backend with remote "gcrypt:"
2022/01/26 16:22:33 DEBUG : Creating backend with remote "GD:crypt"
2022/01/26 16:22:33 DEBUG : Encrypted drive 'gcrypt:': Waiting for checks to finish
2022/01/26 16:22:33 DEBUG : Encrypted drive 'gcrypt:': Waiting for transfers to finish
2022/01/26 16:22:34 DEBUG : hosts: md5 = 6d0770130a65766f2029e6680351164c OK
2022/01/26 16:22:34 INFO  : hosts: Copied (new)
2022/01/26 16:22:34 INFO  :
Transferred:   	        201 B / 201 B, 100%, 0 B/s, ETA -
Transferred:            1 / 1, 100%
Elapsed time:         1.3s

2022/01/26 16:22:34 DEBUG : 7 go routines active

and

2022/01/26 16:23:15 NOTICE: Encrypted drive 'gcrypt:': 0 differences found
2022/01/26 16:23:15 NOTICE: Encrypted drive 'gcrypt:': 1 matching files

and

felix@gemini:~/test$ echo blah >>hosts
felix@gemini:~/test$ rclone cryptcheck /home/felix/test gcrypt:
2022/01/26 16:24:32 ERROR : hosts: Sizes differ
2022/01/26 16:24:32 NOTICE: Encrypted drive 'gcrypt:': 1 differences found
2022/01/26 16:24:32 NOTICE: Encrypted drive 'gcrypt:': 1 errors while checking
2022/01/26 16:24:32 Failed to cryptcheck: 1 differences found
11

agreed, but in the OP case,
rclone copy does not recopy the file, and rclone cryptcheck fails with a mismatched hash.
and what i posted can explain that, was just asking if there was a flaw in my logic?

that rclone copy to a crypt and rclone cryptcheck do not have the same criteria for comparison.
i remember, when first learning about rclone, did the following contrived test.

  1. rclone copy file.txt cryptremote: where file.txt has a size of one byte and timestamp of 01.01.2000.00.00.000
  2. edit file.txt, changing the contents, not the size.
  3. change file.txt timestamp back to 01.01.2020.00.00.000
  4. rclone copy file.txt cryptremote:, this time rclone will not re-copy the file, as size and modtime match the dest.
  5. rclone cryptcheck will fail with a mismatched hash.
12

That's what is being stated but no logs, evidence as that isn't how copy works so back to either OP is wrong, found a bug or is doing something improper.

Your test scenario is accurate but in actual practice, not sure how that scenario would come up unless the OP is setting the clock back or making a huge modify window.

Back to my point I always make, need a log and we spend less time guessing what happened and more time looking at the actual data.

13

good, agreed.

as you can see from this topic, i have been working on that.

14

and just for comedy, you can make a modify window longer and not have to change the timestamp on the file for easier testing:

felix@gemini:~/test$ rclone copy /etc/hosts gcrypt: -vvv  --modify-window 5m
2022/01/26 16:55:55 DEBUG : Setting --config "/opt/rclone/rclone.conf" from environment variable RCLONE_CONFIG="/opt/rclone/rclone.conf"
2022/01/26 16:55:55 DEBUG : rclone: Version "v1.57.0" starting with parameters ["rclone" "copy" "/etc/hosts" "gcrypt:" "-vvv" "--modify-window" "5m"]
2022/01/26 16:55:55 DEBUG : Creating backend with remote "/etc/hosts"
2022/01/26 16:55:55 DEBUG : Using config file from "/opt/rclone/rclone.conf"
2022/01/26 16:55:55 DEBUG : fs cache: adding new entry for parent of "/etc/hosts", "/etc"
2022/01/26 16:55:55 DEBUG : Creating backend with remote "gcrypt:"
2022/01/26 16:55:55 DEBUG : Creating backend with remote "GD:crypt"
2022/01/26 16:55:56 DEBUG : hosts: Size and modification time the same (differ by -28.335µs, within tolerance 5m0s)
2022/01/26 16:55:56 DEBUG : hosts: Unchanged skipping
2022/01/26 16:55:56 INFO  :
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Checks:                 1 / 1, 100%
Elapsed time:         0.7s

2022/01/26 16:55:56 DEBUG : 4 go routines active

I made one letter change in my file which kept the size the same and it does fail to copy.

15

that is good to know

16

See the output from the DEBUG log. There are 4 files in the folder SECURED. 3 check out ok. The 11.07.21 file gives a hash error.

2022/01/26 18:36:55 DEBUG : rclone: Version "v1.57.0" starting with parameters ["rclone" "cryptcheck" "D:\\Docs\\SECURED" "EncryptDocs:SECURED" "-P" "--log-file=Newlog.txt" "--log-level" "DEBUG"]
2022/01/26 18:36:55 DEBUG : Creating backend with remote "D:\\Docs\\SECURED"
2022/01/26 18:37:01 DEBUG : Using config file from "C:\\Users\\Tim\\AppData\\Roaming\\rclone\\rclone.conf"
2022/01/26 18:37:01 DEBUG : fs cache: renaming cache item "D:\\Docs\\SECURED" to be canonical "//?/D:/Docs/SECURED"
2022/01/26 18:37:01 DEBUG : Creating backend with remote "EncryptDocs:SECURED"
2022/01/26 18:37:02 DEBUG : Creating backend with remote "OneDrive:RcloneDocs/v8ielcdf6gveqwhhi1h3iofk4s"
2022/01/26 18:37:03 INFO  : Using sha1 for hash comparisons
2022/01/26 18:37:03 DEBUG : Encrypted drive 'EncryptDocs:SECURED': Waiting for checks to finish
2022/01/26 18:37:04 DEBUG : 2020.pdf: OK
2022/01/26 18:37:04 DEBUG : ReadMe.txt: OK
2022/01/26 18:37:07 ERROR : 11.07.21: hashes differ (EncryptDocs:SECURED) "b45710ccc38fa0e5e7b4f033b378c21ffed7281d" vs (local://?/D:/Docs/SECURED) "280346dd5233fb09030f607591552c76130740c5"
2022/01/26 18:37:07 DEBUG : 10.08.21: OK
2022/01/26 18:37:07 NOTICE: Encrypted drive 'EncryptDocs:SECURED': 1 differences found
2022/01/26 18:37:07 NOTICE: Encrypted drive 'EncryptDocs:SECURED': 1 errors while checking
2022/01/26 18:37:07 NOTICE: Encrypted drive 'EncryptDocs:SECURED': 3 matching files
2022/01/26 18:37:07 INFO  : 
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Errors:                 1 (retrying may help)
Checks:                 4 / 4, 100%
Elapsed time:        12.1s

2022/01/26 18:37:07 DEBUG : 13 go routines active
2022/01/26 18:37:07 Failed to cryptcheck: 1 differences found
17

So only that confirms is they are different.

Can you run the copy, share the full debug and re-run the cryptcheck, share the full output.

18

Sorry, I just deleted that file in OneDrive and re-ran copy. Running cryptcheck again is not giving the hash error. Don't know what caused the hashes to be different. When I was running the copy command earlier it wasn't giving any errors. It was just skipping that file.

19

Ok, so no logs so I assume all is good now?

20

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.