Make --checksum work with crypt remotes

Feature
1

If I get it right from reading https://github.com/ncw/rclone/issues/2903, rclone does not support checksum for crypt remotes. I get this feeling from reading the help text suggestion of the OP:

 -c, --checksum Skip based on checksum & size, not mod-time & size. If checksums don't exist(crypt), then just size.

At first thought it makes sense. After all, the check sum computed by the remote would be completely different than the local checksum. Its encrypted after all.

But after a second thought: why don’t we encrypt the files locally for the only purpose of computing a check sum, and compare this to the remote checksum?

Would this work? Am I missing something?

Cheers,
Eduard

2

I imagine it’s an optimization, because copying and moving operations would potentially take considerably longer if it had the encrypt a whole file just to get the hash. But I could be wrong.

It’s no big issue though, as there is cryptcheck whose only purpose is to verify files in an encrypted remote.

3

Thanks for the hint @AndyIbanezK. But cryptcheck won’t copy files.

If disabling --checksum for sync is a performance optimization, it should be a user decision if the user wants strong consistency or performance.

4

Cryptcheck does exactly this.

The upload process does too - so it computes a hash of the data that is being read and at the end of the upload compares it to the hash produced by the remote end thus making a very strong integrity check.

5

The upload process does too

I see. But it does not so for ‘sync skip check’? To be clear: I talk about a reliable way to verify which files have to be copied and which not.

6

That is correct, --checksum does not work with a crypt backend. If your backend supports times then the modtime + size is a reliable way of determining changes. You can always run rclone cryptcheck to see if it it missed any every now and again.

7

That is correct, --checksum does not work with a crypt backend.

So. What about my suggestion? Would it work?

then the modtime + size is a reliable way

If it would be a reliable way, you would not need --checksum at all. But --checksum is there, and this is for a reason. Because modtime + size is not reliable. I had this case this morning, that rclone sync would not copy anything. But rclone cryptcheck would tell me that there is a file difference. I can’t tell how this happened, but it happened.

8

We could make a --cryptsum flag which would do the equivalent of rclone cryptcheck

  • read the nonce from the remote file
  • encrypt the local file
  • hash the local file

That is a very expensive operation though.

1 Like
9

I believe that is captured here.

EDIT: (I realized I fixed the prior approach I was using so deleted that comment)

10

We could make a --cryptsum flag

That would be great.

That is a very expensive operation though

Sure. Using the flag would be a concisus decision to trade speed for consistency.

11

I put some more ideas on: https://github.com/ncw/rclone/issues/1712 - feel free to add your own comments there.

12

Hey - sorry to bring up an old thread. When I perform an rclone copy from one crypt remote to another, I see Checks 0/0. This doesn't bode confidence that checksums were performed. Could you clarify on whats going on? Thanks

13

You can't use checksums on crypt remotes. Follow the issue that was created and linked above.