Rclone compare two crypt remotes

Help and Support
1

What is the problem you are having with rclone?

Not a problem more a question. Is it possible to compare the contents of two encrypted remotes? Unencrypted data is the same but encrypted with two different sets of passwords. Need to see if there are differences. Trying to move from one remote to another and do a consistency check. I did find Cryptcheck but not sure if it will work with two encrypted remotes that are encrypted with different passwords. Too afraid to try.

Run the command 'rclone version' and share the full output of the command.

rclone v1.62.2

  • os/version: ubuntu 22.04 (64 bit)
  • os/kernel: 5.15.0-72-generic (aarch64)
  • os/type: linux
  • os/arch: arm64 (ARMv8 compatible)
  • go/version: go1.20.2
  • go/linking: static
  • go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Scaleway Object Storage (S3)
Hetzner Storagebox (SFTP)

The command you were trying to run (eg rclone copy /tmp remote:tmp)

Not sure yet

The rclone config contents with secrets removed.

Config is working on both remotes. Checked with “ls” command which lists unencrypted content on both remotes.

A log from the command with the -vv flag

Not available
2

hi,

no fear, as that is just a check, will not damage/delete files

3

Exactly what I am unsure of as well.

4

interesting question. I would compare remote1:crypt1 vs remote2:crypt2. Effectively decrypted content. Assuming both remotes provide the same hash it should be quick.

5

cryptcheck is just a check, will not change files.
so give it a try.

6

@asdffdsa is right - try - it does not write anything. I would try too

7

yes, it is. let's see what the OP finds out. tho tempted to try it myself ;wink

in your case, comparing two decrypted files, would there be any hashes.
the files do not actually exist.

8

yeap. right. after thinking a bit I think cryptcheck is "useless" here. It will download all content from one remote.

9

Some personal hint - what I do for my data is to wrap crypt in chunker with e.g. sha1all - this way I got hashes inside crypt.

10

Running both encrypted remotes against each other at the moment.

Right at the start I got:

2023/05/21 19:34:42 ERROR : dir1/.DS_Store: file not in Encrypted drive 'hetzner-crpt:'

Does that mean it is working?
Still running. Lots of files. Maybe I should have added verbose flag.

11

This is do not fully understand yet…

Could I do this by running cryptcheck against one remote at a time comparing against the “original” data on an external drive (which I also have)?

12

i just did a quick test that worked.

rclone cryptcheck gcrypt01: gcrypt02: -vv

rclone cryptcheck gcrypt01: gcrypt02: -vv 
DEBUG : Setting --config "C:\\data\\rclone\\rclone.conf" from environment variable RCLONE_CONFIG="C:\\data\\rclone\\rclone.conf"
DEBUG : rclone: Version "v1.62.2" starting with parameters ["C:\\data\\rclone\\rclone.exe" "cryptcheck" "gcrypt01:" "gcrypt02:" "-vv"]
DEBUG : Creating backend with remote "gcrypt01:"
DEBUG : Using config file from "C:\\data\\rclone\\rclone.conf"
DEBUG : Creating backend with remote "gdrive:gcrypt01"
DEBUG : Creating backend with remote "gcrypt02:"
DEBUG : Creating backend with remote "gdrive:gcrypt02"
INFO  : Using md5 for hash comparisons
DEBUG : Encrypted drive 'gcrypt02:': Waiting for checks to finish
DEBUG : file.ext: OK
NOTICE: Encrypted drive 'gcrypt02:': 0 differences found
NOTICE: Encrypted drive 'gcrypt02:': 1 matching files
13

I am thinking it does, too. Just have not added -vv so can’t see what it does…

Do you think I can pipe the result in a text file?

14

try this, just need to change the file path to match your system.
--log-level=DEBUG --log-file=/path/to/rclone.log

15

Thank you, makes sense!

Might have a problem with the md5 checksum on the Hetzner Storagebox. Getting a bunch of DEBUG notices.

Will report back again a little later.

Thank you for your help so far!

16

can you post the debug notice?

fwiw, i use storagebox, works fine with md5.
make sure to use port 23, not default 22.

[hetznersbox01_sftp]
type = sftp
host = redacted.your-storagebox.de
port = 23
key_file = C:\data\c\combined\hetzner\sharebox\keys\id_ed25519
user = redacted
shell_type = unix
md5sum_command = md5 -r
sha1sum_command = sha1 -r
17

@Antergosgeek, i think the example below is the way to go for a consistency check,
where the source and dest are both crypt remotes, each with different set of passwords.

the example will download the decrypted contents of both the source and the dest and compare md5 hash
notice that the file to be checked is 1GiB in size and rclone check transferred double that amount at 2GiB

rclone lsl crypt01: 
1073741824 2022-12-06 13:07:35.222000000 1GiB.file

rclone check crypt01: crypt02: --download -vv 
DEBUG : rclone: Version "v1.62.2" starting with parameters ["C:\\data\\rclone\\rclone.exe" "check" "crypt01:" "crypt02:" "--download" "-vv"]
DEBUG : Creating backend with remote "crypt01:"
DEBUG : Creating backend with remote "remote:crypt01"
DEBUG : Creating backend with remote "crypt02:"
DEBUG : Creating backend with remote "remote:crypt02"
DEBUG : Encrypted drive 'crypt02:': Waiting for checks to finish
DEBUG : 1GiB.file: OK
NOTICE: Encrypted drive 'crypt02:': 0 differences found
NOTICE: Encrypted drive 'crypt02:': 1 matching files
INFO  : 
Transferred:   	        2 GiB / 2 GiB, 100%, 40.696 MiB/s, ETA 0s
Checks:                 1 / 1, 100%
Transferred:            1 / 1, 100%
Elapsed time:        52.2s
18

Thank you, means I have to download everything though, correct?

It is about 1 TB :pensive:

19

Interesting, maybe I should add these. Never have.

Thx

20

i would not add them manually as each sftp provider is different.
the first time you connect to a new remote, rclone will set those for you.