Is it possible to run a "reverse" cryptcheck?

[beeper]

I want to run a "reverse" cryptcheck for a remote crypt mount I have vs the source google drive.

The way I understand cryptcheck works after reading the docs is that it first downloads the file in the source Google Drive, encrypts it on the fly using the same nonce from the crypt mount, and then compares the hashes against each other. Instead of that, I want some kind of checksum check where it downloads the encrypted file from the crypt mount, calculates a hash on the unencrypted file, and compares it to the MD5/SHA1 hash already calculated by google drive.

Is this possible? The main reason for this is that my crypt remote has no download bandwidth limitations but the google drive source has a 4-5TB per day download limitation I run into.

[nielash (Niel)]

Not exactly, but there are several pretty good workarounds to accomplish something similar:

1. Hasher backend -- effectively allows crypt to support checksums. See these recent threads for some tips and tricks.

2. rclone hashsum --download --output-file command, combined with check --checkfile. See this post.

3. The latest beta of rclone bisync has a new --download-hash feature that may be of interest.

4. The chunker backend can be used creatively as an alternative to hasher.

5. I actually built the exact feature you're talking about one time... but kind of abandoned the experiment after seeing how slow it would be to use!

@nielash

A few months ago I experimented with making a --crypt-sum flag to force crypt checksum support by generating them on the fly. It worked, but I didn't submit a PR because I figured the Hasher workaround was close enough, and also this would be impractical if you have several TBs of crypted data on cloud remotes (as I do). But your point about the decrypted filenames with hasher is a good one, and perhaps it's enough of a reason to revisit crypt providing hashes itself instead of requiring a (second) overlay. If you're interested, here was my branch that implements this. It is essentially the reverse the approach of "cryptcheck" (instead of re-encrypting the source, it decrypts the dest) which also means it still works when you don't have the source (like if you wanted to compare two different crypt remotes with different passwords.)

(6. I am also now working on something new which could be an interesting solution for this use case, when it's finished.)

[nielash (Niel)]

@beeper

the google drive source has a 4-5TB per day download limitation I run into.

P.S. I so wish my internet was fast enough to have this problem

[beeper]

I see thanks.

One final question - is there a way to run rclone cryptcheck across just a random sample of files comparing one remote to another? For example my remote is 60TB so ideally I'd just run a random sample across say 1TB for cryptcheck and that will tell me if the files transferred generally okay without any issues/corruption.

[nielash (Niel)]

Kind of, but it would take a few steps. For example:

1. Generate a list of files. For example, with:

rclone lsf -R --absolute --files-only remotename: > outputfilename.txt

2. Edit the file to randomly remove some number of lines. There are various tools that can help you with this. You could, for example, sort them in a random order and then copy the first X number of lines to a new file.

3. Use the edited file list as a --files-from filter for cryptcheck.

Alternatively, if true randomness is not important, you could approximate more quickly by getting creative with filters. For example, you could run cryptcheck with something like --include "*q*" to include only filenames which contain the letter q (likely a relatively small sampling -- but not truly random.)

It is an interesting idea and perhaps rclone should consider adding some sort of spot-check command/flag for this use case (for example, I could imagine using it to randomly do a check --download on my files from time to time to look for bit-rot.) If others would find this useful as well, I could probably be convinced to develop this.