Can rclone copy crypted files from one remote to another without decrypting?

Help and Support
41

Okay here are my test results:

rclone copy file1.txt oldRemote: -v
2021/09/09 11:18:50 INFO  : file1.txt: Copied (new)

rclone copy oldRemote:file1.txt newRemote: -v
2021/09/09 11:20:22 INFO  : file1.txt: Copied (new)

rclone lsl file1.txt
2 2021-09-09 11:18:11.315825561 file1.txt

rclone lsl oldRemote: file1.txt
2 2021-09-09 11:18:11.000000000 file1.txt

rclone lsl newRemote file1.txt
2 2021-09-09 11:18:11.000000000 file1.txt

My results differ here, I do not see a time difference in time as you did.

rclone copy newRemote: file1.txt -vv -n
2021/09/09 11:24:01 DEBUG : file1.txt: Modification times differ by 315.825561ms: 2021-09-09 11:18:11 -0400 EDT, 2021-09-09 11:18:11.315825561 -0400 EDT
2021/09/09 11:24:01 NOTICE: file1.txt: Skipped copy as --dry-run is set (size 2)

My results differ here as well, I do not see an md5 reported or a unchanged skipping message

rclone copy newRemote: file1.txt -vv
2021/09/09 11:32:24 DEBUG : file1.txt: Modification times differ by 315.825561ms: 2021-09-09 11:18:11 -0400 EDT, 2021-09-09 11:18:11.315825561 -0400 EDT
2021/09/09 11:32:25 INFO  : file1.txt: Copied (replaced existing)

As indicated in the previous test it copied (downlaoded) the file back to local it did not update the modtime.

42

for the last two commands, post the full debug logs.

43

Here it is:

rclone copy newRemote:folder/file3.txt /folder -vv -n
2021/09/09 12:47:49 DEBUG : rclone: Version "v1.56.0" starting with parameters ["rclone" "copy" "newRemote:file.txt" "/tmp/" "-vv" "-n"]
2021/09/09 12:47:49 DEBUG : Creating backend with remote "kCrypt:/folder/file.txt"
2021/09/09 12:47:49 DEBUG : Using config file from "rclone.conf"
2021/09/09 12:47:49 DEBUG : Creating backend with remote "newRemote:/longstring/longstring2"
2021/09/09 12:47:50 DEBUG : fs cache: adding new entry for parent of "newRemote:/longstring1/longstring2", "newRemote:/longstring1"
2021/09/09 12:47:50 DEBUG : Creating backend with remote "/folder"
2021/09/09 12:47:51 DEBUG : file.txt: Modification times differ by 396.924825ms: 2021-09-09 12:45:06 -0400 EDT, 2021-09-09 12:45:06.396924825 -0400 EDT
2021/09/09 12:47:51 NOTICE: file.txt: Skipped copy as --dry-run is set (size 2)

rclone copy newRemote:folder/file.txt /folder -vv
2021/09/09 12:52:30 DEBUG : rclone: Version "v1.56.0" starting with parameters ["rclone" "copy" "newRemote:file.txt" "/tmp/" "-vv"]
2021/09/09 12:52:30 DEBUG : Creating backend with remote "kCrypt:/folder/file.txt"
2021/09/09 12:52:30 DEBUG : Using config file from "rclone.conf"
2021/09/09 12:52:30 DEBUG : Creating backend with remote "newRemote:/longstring/longstring2"
2021/09/09 12:52:35 DEBUG : fs cache: adding new entry for parent of "newRemote:/longstring1/longstring2", "newRemote:/longstring/longstring1"
2021/09/09 12:52:35 DEBUG : Creating backend with remote "/folder"
2021/09/09 12:52:36 DEBUG : file.txt: Modification times differ by 396.924825ms: 2021-09-09 12:45:06 -0400 EDT, 2021-09-09 12:45:06.396924825 -0400 EDT
2021/09/09 12:52:37 INFO  : file.txt: Copied (replaced existing)
44
  • the command are not the same?
  • the debug logs do not seem to match the commands?
  • the debug logs look strange, are you edtiing them?
45

Yes I redacted the encrypted paths of the remote. I see there is a difference but they were originally the same, I edited the second log incorrectly. I did not delete any lines from the debug log. You can see it did successfully match up and compare the files so I don't think it is a problem with the path. It saw the time offset and copied the file again as opposed to updating the mod time on the source from your test.

I have tested a few times now and it always copies the file for me. I wonder if it is a problem with the hashes?

46

please post full, unedited logs and make they match.

if you need to create a new test case, please do so.

  • create a folder of the root and have the folder contain a single file, so no need to edit long file paths.
  • copy folder to folder

when i saw your command, i did not understand why it not fail with an error.
when i replicated your command, rclone failed with error.

47

I will create a new environment for the tests. I will keep the existing cloud provider remotes but create new crypt wrapper remotes with new passwords which can be deleted after the test so I do not have to redact anything.

This is a step further away from the actual data that I need to work with however but it should at least show us if this basic case matches your results.

48

imho, best to start with a working case, make one change at a time until it fails and then you know where the problem is.

my case worked, i would start with it, exactly matching every detail.
your commands do not match mine.
your newremote does not match mine.

49

Okay, I think I know what is going on. I just repeated you test:

  1. copy file to old remote
  2. copy from from old remote to new remote
  3. lsl local file, old remote, new remote
  4. copy new remote to local
  5. lsl local file, old remote, new remote

The end result is I can reproduce your results, rclone does update the local timestamp and all 3 files ended up with the same timestamp.

Before I proceed to the next step however, will this test extend to a crypted remote? In your base case test, we were just copying over a simple file and we can inspect it at each step.

In the real life use case however I have an encrypted remote. So I need to start by copying a file which gets encrypted to the old remote, and comparing the end result to the unencrypted file in the local filesystem and this is where the test seems to differ and the ends result is it wants to download the file again.

50

after you test, let me know...

51

Preliminary results: rclone copies (downloads) the files locally when wrapping the remotes in a crypt. I will upload the full debug logs later.

52

no need for logs, i replicated it.

53

You replicated my result and see that it does indeed copy (download) the files again when wrapping the remote in a crypt?

54

here is the workaround.

rclone copy crypt:file.txt b:\dest -vv --refresh-times 
DEBUG : rclone: Version "v1.56.0" starting with parameters ["c:\\data\\rclone\\scripts\\rclone.exe" "copy" "crypt:file.txt" "b:\\dest" "-vv" "--refresh-times"]
DEBUG : Creating backend with remote "crypt:file.txt"
DEBUG : Creating backend with remote "remote:crypt/piviejsu06b8311vk99cqb1t6c"
DEBUG : fs cache: adding new entry for parent of "remote:crypt/piviejsu06b8311vk99cqb1t6c", "remote:crypt"
DEBUG : Creating backend with remote "b:\\dest"
DEBUG : fs cache: renaming cache item "b:\\dest" to be canonical "//?/b:/dest"
DEBUG : file.txt: Modification times differ by 981768h0m0s: 2000-01-01 05:00:00 +0000 UTC, 2112-01-01 00:00:00 -0500 EST
INFO  : file.txt: Updated modification time in destination
DEBUG : file.txt: Unchanged skipping
INFO  : 
Transferred:   	          0 / 0 Byte, -, 0 Byte/s, ETA -
Checks:                 1 / 1, 100%
Elapsed time:         0.9s
55

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.