Problem with rclone filename crypt

What is the problem you are having with rclone?

I tired to change the filename encrypt method for encrypted files (base32 encoding to base64) but failed with move command with --crypt-server-side-across-configs. I found it finally results in a download & upload process rather than a "rename".

I further found:
With two configs whose passphase are the same but different filename encrypt method, the hash of the encrypted files are different.
In my mind, since the passphase is the same, the data should be encrpted with the same key and result in same hash but different filename only.

I tested with these two configs:

Passphase - Use the same
Directory encrypt - No (same)
Filename Encrypt - Standard (same)
Encoding - Base64 or Base32 (diff)

Run the command 'rclone version' and share the full output of the command.

➜ ~ rclone version
rclone v1.65.0

  • os/version: arch 23.1.0 (64 bit)
  • os/kernel: 5.15.143-1-MANJARO (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.21.4
  • go/linking: dynamic
  • go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Webdav
I tested the hash on local. So it doesn`t matters.

Webdav supports server side move so you did something wrong. Without details impossible to say more.

This is not how it works... The key used for encryption is permutated with crypto salt every time it is used. So the same file will be encrypted differently. Should actually never be encrypted the same way. It protects from certain class of cryptographic attacks.

If interested you can find plenty of less or more detailed explanations on Internet, e.g.:

So same passphase do not necessarily or say nearly never produce the same key for encrypt use in
Rclone crypt.

Then how could the encrypted file be decrypted? I found password and password2 is different in my generated configs though I type in the same passphrase in configs process.

What will happen if I lost my configs file but remember the original passphase without salt?

If I want to do the change in filename encrypted
method , should I mannual edit the config file to make password and password2 to be the same between two configs? The new base64 one align with the old base32 one.

do some reading.. it is real huge subject. In case of your question - Initialization Vector is not a secret - it is actually part of a file header. So when decrypting you just read it.

It is similar story. passwords encryption (obfuscation really as password is in the source code) is using IV too. So the same password results in different encrypted string.

Your data will remain secret forever:) or until you find salt. But in your case they are both the same. So you just recreate config and all will work. Encrypted value does not really matter. Even if every time different.

What you do is set up a new crypt remote by copying the old one in the config file, renaming it and adding filename_encoding = base64. Choosing a different destination directory is a good idea here too so the old and the new don't overlap.

You can then do a server side move something like

rclone move --server-side-across-configs -vv old: new:

In this case file content will remain the same - only names will change.

Thanks for explaining.

What I got:

In short, the config process will reproduce the same local key.
In encrypting process, IV will be generated and stored in header of new encrypted file.
With IV and the local key, the file can be decrypted.

Since everytime the encrypting process will generated different IVs, the hash of output files are varied with the different combination of local key and IV.

I guess the original hash may be stored in unenrypt part?

Yeap. This is how things work high level indeed.

I am not sure what you mean here...

The different hash of encrypted file -- caused by IVs
The different value of password stored in config file -- obfuscated string (Actually the one will be used still the same one typed in config process)

Both lead me to a wrong direction. :frowning:

I will further check the backend. Maybe something wrong with the WebDAV server.

1 Like

test on smaller dataset. If things do not work post details in the new thread and we will try to figure out what is going on.

To avoid influence of webdav, I tried it locally.
t1 and t2 are the configs created by rclone config with passwd=111 and passwd2=222.

I manully add t3 as a copy of t1 but change the filename encoding.

[local]
type = local

[t1]
type = crypt
remote = local:
directory_name_encryption = false
password = lxiysyKxTX4Pkx1Cp1GfZNdUHw
password2 = _kuqqZhznmqN3VWS5O_aYPdPZQ
filename_encoding = base64

[t2]
type = crypt
remote = local:
directory_name_encryption = false
password = uKoQ1MTK_AXzjCw8yCvei1bX3g
password2 = VkFn558lograObJisw4V6WpUnA
filename_encoding = base32

[t3]
type = crypt
type = crypt
remote = local:
directory_name_encryption = false
password = lxiysyKxTX4Pkx1Cp1GfZNdUHw
password2 = _kuqqZhznmqN3VWS5O_aYPdPZQ
filename_encoding = base32

The working dir is D:\bin.

Step1: create encrypt file under t1 and store in folder 1. and make a backup.

❯ .\rclone copy .\0\testfile.mp4 t1:D:\bin\1 -vv
2023/12/22 13:29:16 DEBUG : rclone: Version "v1.65.0" starting with parameters ["D:\\bin\\rclone.exe" "copy" ".\\0\\testfile.mp4" "t1:D:\\bin\\1" "-vv"]
2023/12/22 13:29:16 DEBUG : Creating backend with remote ".\\0\\testfile.mp4"
2023/12/22 13:29:16 DEBUG : Using config file from "D:\\bin\\rclone.conf"
2023/12/22 13:29:16 DEBUG : fs cache: adding new entry for parent of ".\\0\\testfile.mp4", "//?/D:/bin/0"
2023/12/22 13:29:16 DEBUG : Creating backend with remote "t1:D:\\bin\\1"
2023/12/22 13:29:16 DEBUG : Creating backend with remote "local:D:/bin/n5YSd4p5usa95lJPc1QEQw"
2023/12/22 13:29:16 DEBUG : fs cache: renaming cache item "local:D:/bin/n5YSd4p5usa95lJPc1QEQw" to be canonical "//?/D:/bin/n5YSd4p5usa95lJPc1QEQw"
2023/12/22 13:29:16 DEBUG : Creating backend with remote "local:D:/bin/1"
2023/12/22 13:29:16 DEBUG : fs cache: renaming cache item "local:D:/bin/1" to be canonical "//?/D:/bin/1"
2023/12/22 13:29:16 DEBUG : fs cache: renaming cache item "t1:D:\\bin\\1" to be canonical "t1:D:/bin/1"
2023/12/22 13:29:16 DEBUG : testfile.mp4: Need to transfer - File not found at Destination
2023/12/22 13:30:16 INFO  :
Transferred:        8.543 GiB / 8.541 GiB, 100%, 59.523 MiB/s, ETA -
Transferred:            0 / 1, 0%
Elapsed time:       1m0.1s
Transferring:
 *                                  testfile.mp4:100% /8.541Gi, 59.523Mi/s, -

2023/12/22 13:30:23 DEBUG : testfile.mp4.lihobas3.partial: md5 = 5b464791caebcd902069dd327ee306dc OK
2023/12/22 13:30:23 DEBUG : testfile.mp4.lihobas3.partial: renamed to: testfile.mp4
2023/12/22 13:30:23 INFO  : testfile.mp4: Copied (new)
2023/12/22 13:30:23 INFO  :
Transferred:        8.543 GiB / 8.543 GiB, 100%, 37.886 MiB/s, ETA 0s
Transferred:            1 / 1, 100%
Elapsed time:       1m7.3s

2023/12/22 13:30:23 DEBUG : 5 go routines active

Step2: Move it from t1 to t2:

❯ .\rclone move t1:D:\bin\1\ t2:D:\bin\2 -vv --crypt-server-side-across-configs
2023/12/22 13:32:27 DEBUG : rclone: Version "v1.65.0" starting with parameters ["D:\\bin\\rclone.exe" "move" "t1:D:\\bin\\1\\" "t2:D:\\bin\\2" "-vv" "--crypt-server-side-across-configs"]
2023/12/22 13:32:27 DEBUG : Creating backend with remote "t1:D:\\bin\\1\\"
2023/12/22 13:32:27 DEBUG : Using config file from "D:\\bin\\rclone.conf"
2023/12/22 13:32:27 DEBUG : t1: detected overridden config - adding "{Db_Y9}" suffix to name
2023/12/22 13:32:27 DEBUG : Creating backend with remote "local:D:/bin/1"
2023/12/22 13:32:27 DEBUG : fs cache: renaming cache item "local:D:/bin/1" to be canonical "//?/D:/bin/1"
2023/12/22 13:32:27 DEBUG : fs cache: switching user supplied name "local:D:/bin/1" for canonical name "//?/D:/bin/1"
2023/12/22 13:32:27 DEBUG : fs cache: renaming cache item "t1:D:\\bin\\1\\" to be canonical "t1{Db_Y9}:D:/bin/1/"
2023/12/22 13:32:27 DEBUG : Creating backend with remote "t2:D:\\bin\\2"
2023/12/22 13:32:27 DEBUG : t2: detected overridden config - adding "{Db_Y9}" suffix to name
2023/12/22 13:32:27 DEBUG : Creating backend with remote "local:D:/bin/ro0qvh2oqj1j674hv6s0ott710"
2023/12/22 13:32:27 DEBUG : fs cache: renaming cache item "local:D:/bin/ro0qvh2oqj1j674hv6s0ott710" to be canonical "//?/D:/bin/ro0qvh2oqj1j674hv6s0ott710"
2023/12/22 13:32:27 DEBUG : Creating backend with remote "local:D:/bin/2"
2023/12/22 13:32:27 DEBUG : fs cache: renaming cache item "local:D:/bin/2" to be canonical "//?/D:/bin/2"
2023/12/22 13:32:27 DEBUG : fs cache: renaming cache item "t2:D:\\bin\\2" to be canonical "t2{Db_Y9}:D:/bin/2"
2023/12/22 13:32:27 DEBUG : testfile.mp4: Need to transfer - File not found at Destination
2023/12/22 13:32:27 DEBUG : Encrypted drive 't2{Db_Y9}:D:/bin/2': Waiting for checks to finish
2023/12/22 13:32:27 DEBUG : Encrypted drive 't2{Db_Y9}:D:/bin/2': Waiting for transfers to finish
2023/12/22 13:33:27 INFO  :
Transferred:        8.543 GiB / 8.541 GiB, 100%, 78.352 MiB/s, ETA -
Checks:                 0 / 1, 0%
Transferred:            0 / 1, 0%
Elapsed time:       1m0.1s
Checking:

Transferring:
 *                                  testfile.mp4:100% /8.541Gi, 78.354Mi/s, -

2023/12/22 13:33:38 DEBUG : testfile.mp4.nokipit2.partial: md5 = ae41a74a9d75b0b385ee5502cb7d371a OK
2023/12/22 13:33:38 DEBUG : testfile.mp4.nokipit2.partial: renamed to: testfile.mp4
2023/12/22 13:33:38 INFO  : testfile.mp4: Copied (new)
2023/12/22 13:33:38 INFO  : testfile.mp4: Deleted
2023/12/22 13:33:38 INFO  :
Transferred:        8.543 GiB / 8.543 GiB, 100%, 36.117 MiB/s, ETA 0s
Checks:                 2 / 2, 100%
Deleted:                1 (files), 0 (dirs)
Renamed:                1
Transferred:            1 / 1, 100%
Elapsed time:      1m11.6s

2023/12/22 13:33:38 DEBUG : 5 go routines active

Step3: Move it from t1 to t3:

❯ .\rclone move t1:D:\bin\1\ t3:D:\bin\3 -vv --crypt-server-side-across-configs
2023/12/22 13:42:49 DEBUG : rclone: Version "v1.65.0" starting with parameters ["D:\\bin\\rclone.exe" "move" "t1:D:\\bin\\1\\" "t3:D:\\bin\\3" "-vv" "--crypt-server-side-across-configs"]
2023/12/22 13:42:49 DEBUG : Creating backend with remote "t1:D:\\bin\\1\\"
2023/12/22 13:42:49 DEBUG : Using config file from "D:\\bin\\rclone.conf"
2023/12/22 13:42:49 DEBUG : t1: detected overridden config - adding "{Db_Y9}" suffix to name
2023/12/22 13:42:49 DEBUG : Creating backend with remote "local:D:/bin/1"
2023/12/22 13:42:49 DEBUG : fs cache: renaming cache item "local:D:/bin/1" to be canonical "//?/D:/bin/1"
2023/12/22 13:42:49 DEBUG : fs cache: switching user supplied name "local:D:/bin/1" for canonical name "//?/D:/bin/1"
2023/12/22 13:42:49 DEBUG : fs cache: renaming cache item "t1:D:\\bin\\1\\" to be canonical "t1{Db_Y9}:D:/bin/1/"
2023/12/22 13:42:49 DEBUG : Creating backend with remote "t3:D:\\bin\\3"
2023/12/22 13:42:49 DEBUG : t3: detected overridden config - adding "{Db_Y9}" suffix to name
2023/12/22 13:42:49 DEBUG : Creating backend with remote "local:D:/bin/sr9lnttqj00cesvl9rb8o9khjo"
2023/12/22 13:42:49 DEBUG : fs cache: renaming cache item "local:D:/bin/sr9lnttqj00cesvl9rb8o9khjo" to be canonical "//?/D:/bin/sr9lnttqj00cesvl9rb8o9khjo"
2023/12/22 13:42:49 DEBUG : Creating backend with remote "local:D:/bin/3"
2023/12/22 13:42:49 DEBUG : fs cache: renaming cache item "local:D:/bin/3" to be canonical "//?/D:/bin/3"
2023/12/22 13:42:49 DEBUG : fs cache: renaming cache item "t3:D:\\bin\\3" to be canonical "t3{Db_Y9}:D:/bin/3"
2023/12/22 13:42:49 DEBUG : testfile.mp4: Need to transfer - File not found at Destination
2023/12/22 13:42:49 DEBUG : Encrypted drive 't3{Db_Y9}:D:/bin/3': Waiting for checks to finish
2023/12/22 13:42:49 DEBUG : Encrypted drive 't3{Db_Y9}:D:/bin/3': Waiting for transfers to finish
2023/12/22 13:43:49 INFO  :
Transferred:        8.543 GiB / 8.541 GiB, 100%, 79.461 MiB/s, ETA -
Checks:                 0 / 1, 0%
Transferred:            0 / 1, 0%
Elapsed time:       1m0.1s
Checking:

Transferring:
 *                                  testfile.mp4:100% /8.541Gi, 84.788Mi/s, -

2023/12/22 13:44:03 DEBUG : testfile.mp4.yupexij1.partial: md5 = f15082dd876099ec14367bde7c522bae OK
2023/12/22 13:44:03 DEBUG : testfile.mp4.yupexij1.partial: renamed to: testfile.mp4
2023/12/22 13:44:03 INFO  : testfile.mp4: Copied (new)
2023/12/22 13:44:03 INFO  : testfile.mp4: Deleted
2023/12/22 13:44:03 INFO  :
Transferred:        8.543 GiB / 8.543 GiB, 100%, 32.192 MiB/s, ETA 0s
Checks:                 2 / 2, 100%
Deleted:                1 (files), 0 (dirs)
Renamed:                1
Transferred:            1 / 1, 100%
Elapsed time:      1m14.1s

2023/12/22 13:44:03 DEBUG : 5 go routines active

Both in step2 and step3, I expect a "rename" and move in file operation.
But it seems doing a data decrypt and encrypt in the process, based on the operation time cost.

Is there any misunderstanding?

Use --server-side-across-configs and not --crypt-server-side-across-configs flag which is deprecated as per docs

Not sure if different obfuscated passwords matter (I know you use the same password everywhere).

I do not have Windows to replicate exactly what you are doing but see my example:

[onedrive]
type = onedrive
drive_id = XXX
drive_type = personal
token = XXX

[base32remote]
type = crypt
remote = onedrive:base32
password = lxiysyKxTX4Pkx1Cp1GfZNdUHw
password2 = _kuqqZhznmqN3VWS5O_aYPdPZQ

[base64remote]
type = crypt
# please note different folder used than for base32remote
remote = onedrive:base64
password = lxiysyKxTX4Pkx1Cp1GfZNdUHw
password2 = _kuqqZhznmqN3VWS5O_aYPdPZQ
filename_encoding = base64

there is one test file in base32 remote:

$ rclone lsf onedrive:base32
cj601515p2sf4fkhmhtjp3ooqk

$ rclone lsf base32remote:
test.file

I move it server side - you can see in the log that INFO : test.file: Moved (server-side):

$ rclone move base32remote: base64remote: --server-side-across-configs -vv
2023/12/22 08:34:20 DEBUG : rclone: Version "v1.65.0" starting with parameters ["rclone" "move" "base32remote:" "base64remote:" "--server-side-across-configs" "-vv"]
2023/12/22 08:34:20 DEBUG : Creating backend with remote "base32remote:"
2023/12/22 08:34:20 DEBUG : Using config file from "/Users/kptsky/.config/rclone/rclone.conf"
2023/12/22 08:34:20 DEBUG : Creating backend with remote "onedrive:base32"
2023/12/22 08:34:20 DEBUG : Creating backend with remote "base64remote:"
2023/12/22 08:34:20 DEBUG : Creating backend with remote "onedrive:base64"
2023/12/22 08:34:21 DEBUG : test.file: Need to transfer - File not found at Destination
2023/12/22 08:34:21 DEBUG : Encrypted drive 'base64remote:': Waiting for checks to finish
2023/12/22 08:34:21 DEBUG : Encrypted drive 'base64remote:': Waiting for transfers to finish
2023/12/22 08:34:24 INFO  : test.file: Moved (server-side)
2023/12/22 08:34:24 INFO  : There was nothing to transfer
2023/12/22 08:34:24 INFO  :
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Checks:                 1 / 1, 100%
Renamed:                1
Server Side Moves:      1 @ 419.212 MiB
Elapsed time:         4.3s

2023/12/22 08:34:24 DEBUG : 11 go routines active

and now its name is base64 encoded:

$ rclone lsf onedrive:base64
ZMsAzCXIuPI-kbR7TJcY1Q

$ rclone lsf base64remote:
test.file

On the other note. Are you sure you can use base64 on your Webdav? Is it case sensitive?

can you create two files like:

aaa.txt
AAA.txt

I'm sure it is case sensitive.
I will try the new server side flag.

--server-side-across-configs works well
The failure was caused by deprecated flag. :sob:

1 Like

To get even better results in terms of path length you can try base32768.

Test it first by running:

rclone test info --check-length --check-base32768 remote:test/folder

sample results for remote supporting this encoding:

$ rclone test info --check-length --check-base32768 .
// local
maxFileLength = 255 // for 1 byte unicode characters
maxFileLength = 255 // for 2 byte unicode characters
maxFileLength = 255 // for 3 byte unicode characters
maxFileLength = 127 // for 4 byte unicode characters
base32768isOK = true // make sure maxFileLength for 2 byte unicode chars is the same as for 1 byte characters

I got 512 length for first two and false for the last.
Base64 might be the best option.

yeap - if base32768isOK = false then you can't use it.

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.