Can rclone copy crypted files from one remote to another without decrypting?

Help and Support
21

Okay I understand why I confused you now. Local has unencrypted file, newRemote has crypted files. I was thinking I could compare the remote (crypted) with local (unencrypted) using check and obviously this will not work.

So it sounds like:
rclone cryptcheck localFileSystem: newRemote:
is exactly what i was looking for.

From my understanding this will looks at the local file and compared that hash to the hash stored in the crypt file metadata header, is this correct?

Sorry for the confusion but from the discussion you again really helped me discover this new command.

22

well, rclone needs two items from the crypt file.

  • the hash stored as metadata
  • the nonce read from the file itself.

but based on what you wrote,
i though that newRemote: contains files that are crypted, but itself is not a crypt remote?

23

This is correct, newRemote is a plain Koofr remote using an oauth token provided from the Koofr web interface so it holds the crypted files. It isn't a crypt wrapper remote however.

Oh I see so cryptcheck only works against a crypted remote. So when I am finished, I can wrap newRemote in a crypt remote (using the same password as my old crypt remote) and I have two options:

rclone cryptcheck newRemote: newCryptRemote:

I think this may work because newRemote: supports MD5 and the crypt files have an MD5 stored inside them.

or

rclone cryptcheck /local/files newCryptRemote:

I think this will work because the local files can be hashed with MD5 to match the hash stored in the crypt file.

24

i would not trust that.
that a file could have been corrupted during the copy from oldRemote: to newRemote:
that command might not catch that.

given that you have a local copy, i use cryptcheck against that.

25

Okay i will try this. The transfer has completed and I have a new issue. Several files were returning errors when copying. Here is an example:

2021-09-04 20:29:58 ERROR : path/path/path/pathpath/path/looooongfilename: Failed to copy: Invalid response status! Got 400, expected [200]; headers: map[Content-Length:[0] Content-Type:[text/plain] Date:[Sat, 04 Sep 2021 22:29:58 GMT] X-Request-Id:[xxxxxx] X-User-Id:[xxxxxxx]], content:

I will open a new thread with a proper debug log to verify however I contacted Koofr support and they seem to know exactly what the problem is:

From Koofr support:

The errors you are seeing are a result of too long filenames, due to us supporting various platforms we had to limit filenames to 255 characters. If you are using rclone you can set a limit for the filename length in the settings.

I am not sure if this is the cause however because according to this: Crypt 146 is the max file length for crypted files. One of the files that is throwing an error has a total path length of 578 characters, the actual filename itself is 282 characters. So I am not even sure how that file got there if crypt limits filenames are limited to 146 characters.

I will get a debug log but in the meantime, are you aware of the max file size setting they mentioned? Is there a --max-file-size flag or something similar? I didn't see anything here: Global Flags or here Koofr

26

did you ever post the operating system or rclone version?

well, ~143 characters which might mean that is not a hard limit.
perhaps certain providers allow for filenames longer that 255.
you never posted the config file, so no idea what the storage provider is for the source remote???

i think you mean max file length
i know of no such flag and even if it existed, what would be its use?

  • ask them for the exact rclone setting?
  • ask them if is a max length for the entire path, not just the filename?
27

maybe this is what koofr had in mind
https://rclone.org/local/#long-paths

28

Here is the version:
Rclone version:
rclone v1.56.0

  • os/version: ubuntu 20.04 (64 bit)
  • os/kernel: 5.4.0-73-generic (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.16.5
  • go/linking: static
  • go/tags: none

I downloaded the .deb from the rclone website and installed it on the VPS.

Regarding the config file, here is the Koofr part:

[newRemote]
type = koofr
user = user@domain.com
password = password

The source is similar however I would be more comfortable waiting until the data is safely transferred and validated.

Asked, I will see what they say.

I ran a new rclone sync and the debug log is very simple because there are no transfers, everything is synced already except those error files and the debug errors are the same one line entry I posted earlier.

2021-09-04 20:29:58 ERROR : path/path/path/pathpath/path/looooongfilename: Failed to copy: Invalid response status! Got 400, expected [200]; headers: map[Content-Length:[0] Content-Type:[text/plain] Date:[Sat, 04 Sep 2021 22:29:58 GMT] X-Request-Id:[xxxxxx] X-User-Id:[xxxxxxx]], content:

It might be helpful to know what status 400 and 200 are. I assume this specific error is coming from the Koofr API and rclone is just reporting it.

29
  1. rclone makes a request to koofr
  2. f koofr accepts the request, koofr will reply with status code 200.
  3. if koofr rejects the the request, it will reply with an error number, in this case 400

https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/200
https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/400

for a more detailed log, use
--dump=headers --retries=1 --low-level-retries=1 --log-level=DEBUG --log-file=rclone.log

30

Thank I will try the --dump=headers flag next.

Some interesting preliminary results from the rclone cryptcheck /local/files newRemoteCrypt:

2021/09/04 17:56:36 ERROR : .path/file hashes differ (newRemoteCrypt) "hash1" vs (local:/path/) "hash2"

A small number of files so far but I didn't actually expect to see any. Bitrot? The local files seem to be intact although I didn't know what their MD5 hash was before so I can't tell if the local file changed or the remote version. This is from a backup data set so not live files that would have changed. Interesting results, thanks for the info on the cryptcheck operation.

31

compare the local file against the crypted file in oldRemoteCrypt:

32

I have cleaned up all of the crypted file names that has paths over the character limit at it seemed to fix the problem.

I think I have a major problem. I tried to do a rclone sync /local/file newRemoteCrypt:

The sync is able to detect and match up the files. The problem is that even though the mod times are the same (within a second difference), rclone still tries to transfer the file! Here is the command I tried:

rclone --dry-run -vv /path/testFile kCrypt:path/testFile

Here is the log:

2021/09/05 17:55:39 DEBUG : testFile: Modification times differ by -451.489ms: 2019-02-13 00:17:54.451489 -0500 EST, 2019-02-13 00:17:54 -0500 EST

2021/09/05 17:55:39 NOTICE: testFile: Skipped copy as --dry-run is set (size 7.812Mi)

Rclone recognizes time offset is less than a second however it does not behave as expected. It does not report the difference as being inside the 1s tolerance it reports the times differ and it still tries to copy the file.

Here is the same test with the old cloud provider:

2021/09/05 18:03:26 DEBUG : testFile: Size and modification time the same (differ by -482.1539ms, within tolerance 1s)

2021/09/05 18:03:26 DEBUG : testFile: Unchanged skipping

Rclone reports a similar offset but recognizes it is within tollerance and skips copying, this is as expected.

I double checked, the file sizes are identical and the mod time are within a second. This obviously breaks sync and the whole journey of trying to migrate the data without re-uploading.

According to this: Koofr does support modtimes and it is set to true by default.

Any ideas on how to fix this?

33

each provider deals with with modtime differently.
since you have not revealed the super secret source provider, hard to tell you more.
based on the post i am going to guess the source is onedrive, box or pcloud, which use sha1 and modtime resolution of 1s.

https://rclone.org/docs/#modify-window-time

34

Yes you are correct the source I am migrating away from is one of those providers.

So with this in mind do you know what is going on? Rclone is able to read and parse the modtime of the file correctly and knows it is less than 1 second difference, so why does it decide it is not within tolerance and tries to copy the file again?

I really thought we could get this to work, all the tools appeared to be there in place and I REALLY appreciated all you help with this. This is basically the last step of the whole journey.

35

yes, i know what is going on.

tl;dr - try --modify-window=1s

36

Okay I think I have it!!!! I set --modify-window=60s

after I read:

The default is 1ns unless this is overridden by a remote.

Because I have only ever had experience with a single cloud provider, I thought the default was 1m... I also went searching for a flag before I posed, I was actually was looking at Global Flags and searching for "modtime" or "modification" etc. but didn't see anything.

My sync appears to be working, this is the last piece of the puzzle. I am syncing from local now, the journal is over.

Do you think I should change the title of the post? This conversation basically basically a guide for migrating data from one host to another. Perhaps I can make it easier for others to find in the future. You linked to so many necessary and useful tools and also provided a narrative to their use that ties the whole process together. One area where I and perhaps other newer rcloners struggle with is discoverability. I search for flags that I think should exist but either you have to know the precise name or like this last one they are documented in different pages.

Your knowledge of rclone is as impressive as the rclone project itself. Thank you very much.

37

my ego thanks you very much and in repay, it might have found a permanent solution to the problem.

  • no need to use --modify-window and forever deal with modtime mismatches.
  • if you were to forget --modify-window, then that would trigger rclone to re-copy every single file.
    my freshly stroked ego will not let that happen to you.

tl;dr - the permanent solution
a one time copy from newremote to local, rclone will see the modtime mismatch.
rclone will be able to change the modtime on local to match the modtime on newremote, without triggering a re-copy.
after that rclone copy from local to newremote, the modtimes will match and no need to worry about forgetting --modify-window

for me, i found this an excellent use-case, with many moving parts; i enjoyed working thru it.

  • three storage systems
    --- local
    --- oldremote: which is one of the following onedrive, box or pcloud
    --- newremote: koofr

  • three modtime window resolutions,
    --- 10ns - local on windows, 1ns on linux.
    --- 1s - oldremote -> which is one of the following onedrive, box or pcloud
    --- 1ms - newremote -> koofr

  • two hash types
    --- sha1/md5 - local
    --- sha1 - oldremote -> which is one of the following onedrive, box or pcloud
    --- md5 - newremote -> koofr

  • multiple copy operations, with different modtime resolutions
    --- local to oldremote -> 10ns to 1s
    --- oldremote to newremote -> 1s to 1ms
    --- local to newremote -> 1ms to 10ns

in the end, we were left with a set of files on local and newremote.
--- same hashes
--- different modtimes

the workaround is to use --modify-window.

here is your current situation, copying a file from local to newremote and dealing with the modtime mismatch

with --dry-run - rclone sees that the hashes match, but modtime is different, rclone wants to update modtime

rclone copy file.txt newremote: -vv --dry-run 
DEBUG : file.txt: Modification times differ by -759.2967ms: 2021-08-24 16:10:57.7592967 -0400 EDT, 2021-08-24 16:10:57 -0400 EDT
DEBUG : file.txt: md5 = 898a314f5208bb3d0eec9280c3793acf OK
NOTICE: file.txt: Skipped update modification time as --dry-run is set (size 275)
DEBUG : file.txt: Unchanged skipping

without --dry-run - same as last test - rclone sees that the hashes match, but modtime is different, rclone wants to update modtime.
time to cue the scary music...
but with koofr, once a file is uploaded, the modtime cannot be changed without re-copying the file.
rclone will re-copy the file to get modtimes to match,

rclone copy file.txt newremote: -vv 
DEBUG : file.txt: Modification times differ by -759.2967ms: 2021-08-24 16:10:57.7592967 -0400 EDT, 2021-08-24 16:10:57 -0400 EDT
DEBUG : file.txt: md5 = 898a314f5208bb3d0eec9280c3793acf OK
NOTICE: koofr:c6c75a7c-f8ae-0044-bb82-f74bb377d285:testfolder01: Forced to upload files to set modification times on this backend.
INFO  : file.txt: src and dst identical but can't set mod time without deleting and re-uploading
DEBUG : file.txt: md5 = 898a314f5208bb3d0eec9280c3793acf OK
INFO  : file.txt: Copied (replaced existing)

here are the test i did to confirm the permanent solution.

rclone delete newremote:testfolder01 -v 
2021/09/06 16:01:59 INFO  : file.txt: Deleted

rclone delete oldremote:testfolder01 -v 
2021/09/06 16:02:02 INFO  : file.txt: Deleted

rclone copy file.txt oldremote:testfolder01 -v 
2021/09/06 16:02:07 INFO  : file.txt: Copied (new)

rclone copy oldremote:testfolder01 newremote:testfolder01 -v 
2021/09/06 16:02:10 INFO  : file.txt: Copied (new)

rclone lsl file.txt 
      275 2000-01-01 01:01:01.100000000 file.txt

rclone lsl oldremote:testfolder01 
      275 2000-01-01 01:01:01.000000000 file.txt

rclone lsl newremote:testfolder01 
      275 2000-01-01 01:01:01.000000000 file.txt

rclone copy newremote:testfolder01 .\source -vv --dry-run 
DEBUG : file.txt: Modification times differ by 100ms: 2000-01-01 01:01:01 -0500 EST, 2000-01-01 01:01:01.1 -0500 EST
DEBUG : file.txt: md5 = 898a314f5208bb3d0eec9280c3793acf OK
NOTICE: file.txt: Skipped update modification time as --dry-run is set (size 275)
DEBUG : file.txt: Unchanged skipping

rclone copy newremote:testfolder01 .\source -vv 
DEBUG : file.txt: Modification times differ by 100ms: 2000-01-01 01:01:01 -0500 EST, 2000-01-01 01:01:01.1 -0500 EST
DEBUG : file.txt: md5 = 898a314f5208bb3d0eec9280c3793acf OK
INFO  : file.txt: Updated modification time in destination
DEBUG : file.txt: Unchanged skipping

rclone copy newremote:testfolder01 .\source -vv 
DEBUG : file.txt: Size and modification time the same (differ by 0s, within tolerance 1ms)
DEBUG : file.txt: Unchanged skipping
INFO  : There was nothing to transfer

rclone lsl file.txt 
      275 2000-01-01 01:01:01.000000000 file.txt

rclone lsl oldremote:testfolder01 
      275 2000-01-01 01:01:01.000000000 file.txt

rclone lsl newremote:testfolder01 
      275 2000-01-01 01:01:01.000000000 file.txt
38

This is very interesting. Does the one time copy actually copy (download) the files from newremote to local or does it just update the modtime on the local file?

From your test it looks like it didn't actually use bandwidth and download the file but rather just updated the modtime.

rclone copy newremote:testfolder01 .\source -vv --dry-run 
DEBUG : file.txt: Modification times differ by 100ms: 2000-01-01 01:01:01 -0500 EST, 2000-01-01 01:01:01.1 -0500 EST
DEBUG : file.txt: md5 = 898a314f5208bb3d0eec9280c3793acf OK
NOTICE: file.txt: Skipped update modification time as --dry-run is set (size 275)
DEBUG : file.txt: Unchanged skipping

rclone copy newremote:testfolder01 .\source -vv 
DEBUG : file.txt: Modification times differ by 100ms: 2000-01-01 01:01:01 -0500 EST, 2000-01-01 01:01:01.1 -0500 EST
DEBUG : file.txt: md5 = 898a314f5208bb3d0eec9280c3793acf OK
INFO  : file.txt: Updated modification time in destination
DEBUG : file.txt: Unchanged skipping

Am I understanding this correctly? I tried an initial recreation of your test however it did actually download the file.

39

if the same file exists in newremote and local with

  • same hash
  • different modtime
    then rclone will change the modtime of local file without a download.

would need to see the full debug log.

40

Wow this is pretty powerful. I will try to recreate your test step by step and post my results. It is likely I am missing a piece. Thank you.