Different cache dir suffixes for same password

Help and Support
1

What is the problem you are having with rclone?

I do not want to store my passwords and salts on disk,
so I am using environment variables to supply these on the fly
like this:

export RCLONE_CRYPT_PASSWORD1=$(command-producing-password | rclone obscure -)
export RCLONE_CRYPT_PASSWORD2=$(command-producing-salt | rclone obscure -)

The actual passwords come from my password manager.

Using environment variables triggers a message:

detected overridden config - adding "{xxxxx}" suffix to name

Problem is: this suffix is different every time I mount, which causes two problems:

  1. A lot of vfs/storagebox-crypt{xxxxx} dirs build up in the cache directory
    that take up space and never get cleaned up.
  2. Not-yet-uploaded files waiting in those dirs that never get 'resumed'
    so won't get written to the remote.

I'm guessing the suffix is based on the hash of the obscured salt,
which is different every time I run rclone obscure.
I order to prevent this I would have to store the obscured password in my password manager, next to the actual password.
I guess I could do that, but I'm wondering if this is the correct solution.

I can think of the following solutions on the rclone side:

  1. Don't hash obscured passwords and salts, but 'decrypt' them first.
  2. Don't hash passwords at all and ignore them when producing the suffix.
  3. Don't require obfuscation of passwords and salts in environment variables. This would be difficult to do while being backwards compatible.
  4. Make rclone obscure deterministic.

Option 3 would be more of a workaround.

A combination of 1 and 2 might be needed: I'm guessing a changed salt
doesn't have an influence on accessing existing encrypted content
but a changed password does.

Run the command 'rclone version' and share the full output of the command.

rclone 1.62.2
- os/version: nixos 23.05 (Stoat) (64 bit)
- os/kernel: 6.1.38 (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.20.6
- go/linking: dynamic
- go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

Webdav (Hetzner storagebox) + crypt

The command you were trying to run (eg rclone copy /tmp remote:tmp)

Obtaining the actual salt is left out here.
Here I'm directly passing the salt to make things simpler:

export RCLONE_CRYPT_PASSWORD2=$(rclone obscure xxxxxxxxxxxxx)
rclone mount -vv --config=rclone.conf --vfs-cache-mode writes storagebox-crypt: crypt-mount/

The rclone config contents with secrets removed.

[storagebox]
type = webdav
url = https://xxxxx.your-storagebox.de
vendor = other
user = xxxxx
pass = xxxxx

[storagebox-crypt]
type = crypt
remote = storagebox:probeer
password = xxxxx
password2 = xxxxx

A log from the command with the -vv flag

Run 1:

2023/08/19 20:36:49 DEBUG : Setting default for crypt-password2="xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" from environment variable RCLONE_CRYPT_PASSWORD2
2023/08/19 20:36:49 DEBUG : rclone: Version "1.62.2" starting with parameters ["/run/current-system/sw/bin/rclone" "mount" "-vv" "--config=rclone.conf" "--vfs-cache-mode" "writes" "storagebox-crypt:" "crypt-mount/"]
2023/08/19 20:36:49 DEBUG : Creating backend with remote "storagebox-crypt:"
2023/08/19 20:36:49 DEBUG : Using config file from "/home/jeroen/meuk/rclone-gocryptfs/rclone.conf"
2023/08/19 20:36:49 DEBUG : Setting crypt_password2="xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" from environment variable RCLONE_CRYPT_PASSWORD2
2023/08/19 20:36:49 DEBUG : storagebox-crypt: detected overridden config - adding "{84U5l}" suffix to name
2023/08/19 20:36:49 DEBUG : Setting crypt_password2="xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" from environment variable RCLONE_CRYPT_PASSWORD2
2023/08/19 20:36:50 DEBUG : Creating backend with remote "storagebox:probeer"
2023/08/19 20:36:50 DEBUG : found headers: 
2023/08/19 20:36:50 DEBUG : fs cache: renaming cache item "storagebox-crypt:" to be canonical "storagebox-crypt{84U5l}:"
2023/08/19 20:36:50 INFO  : Encrypted drive 'storagebox-crypt{84U5l}:': poll-interval is not supported by this remote
2023/08/19 20:36:50 DEBUG : vfs cache: root is "/home/jeroen/.cache/rclone"
2023/08/19 20:36:50 DEBUG : vfs cache: data root is "/home/jeroen/.cache/rclone/vfs/storagebox-crypt{84U5l}"
2023/08/19 20:36:50 DEBUG : vfs cache: metadata root is "/home/jeroen/.cache/rclone/vfsMeta/storagebox-crypt{84U5l}"
2023/08/19 20:36:50 DEBUG : Creating backend with remote "/home/jeroen/.cache/rclone/vfs/storagebox-crypt{84U5l}/"
2023/08/19 20:36:50 DEBUG : fs cache: renaming cache item "/home/jeroen/.cache/rclone/vfs/storagebox-crypt{84U5l}/" to be canonical "/home/jeroen/.cache/rclone/vfs/storagebox-crypt{84U5l}"
2023/08/19 20:36:50 DEBUG : Creating backend with remote "/home/jeroen/.cache/rclone/vfsMeta/storagebox-crypt{84U5l}/"
2023/08/19 20:36:50 DEBUG : fs cache: renaming cache item "/home/jeroen/.cache/rclone/vfsMeta/storagebox-crypt{84U5l}/" to be canonical "/home/jeroen/.cache/rclone/vfsMeta/storagebox-crypt{84U5l}"
2023/08/19 20:36:50 INFO  : vfs cache: cleaned: objects 0 (was 0) in use 0, to upload 0, uploading 0, total size 0 (was 0)
2023/08/19 20:36:50 DEBUG : Encrypted drive 'storagebox-crypt{84U5l}:': Mounting on "crypt-mount/"

Run 2, different suffix:

2023/08/19 20:40:02 DEBUG : Setting default for crypt-password2="xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" from environment variable RCLONE_CRYPT_PASSWORD2
2023/08/19 20:40:02 DEBUG : rclone: Version "1.62.2" starting with parameters ["/run/current-system/sw/bin/rclone" "mount" "-vv" "--config=rclone.conf" "--vfs-cache-mode" "writes" "storagebox-crypt:" "crypt-mount/"]
2023/08/19 20:40:02 DEBUG : Creating backend with remote "storagebox-crypt:"
2023/08/19 20:40:02 DEBUG : Using config file from "/home/jeroen/meuk/rclone-gocryptfs/rclone.conf"
2023/08/19 20:40:02 DEBUG : Setting crypt_password2="xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" from environment variable RCLONE_CRYPT_PASSWORD2
2023/08/19 20:40:02 DEBUG : storagebox-crypt: detected overridden config - adding "{gy3BQ}" suffix to name
2023/08/19 20:40:02 DEBUG : Setting crypt_password2="xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" from environment variable RCLONE_CRYPT_PASSWORD2
2023/08/19 20:40:02 DEBUG : Creating backend with remote "storagebox:probeer"
2023/08/19 20:40:02 DEBUG : found headers: 
2023/08/19 20:40:02 DEBUG : fs cache: renaming cache item "storagebox-crypt:" to be canonical "storagebox-crypt{gy3BQ}:"
2023/08/19 20:40:02 INFO  : Encrypted drive 'storagebox-crypt{gy3BQ}:': poll-interval is not supported by this remote
2023/08/19 20:40:02 DEBUG : vfs cache: root is "/home/jeroen/.cache/rclone"
2023/08/19 20:40:02 DEBUG : vfs cache: data root is "/home/jeroen/.cache/rclone/vfs/storagebox-crypt{gy3BQ}"
2023/08/19 20:40:02 DEBUG : vfs cache: metadata root is "/home/jeroen/.cache/rclone/vfsMeta/storagebox-crypt{gy3BQ}"
2023/08/19 20:40:02 DEBUG : Creating backend with remote "/home/jeroen/.cache/rclone/vfs/storagebox-crypt{gy3BQ}/"
2023/08/19 20:40:02 DEBUG : fs cache: renaming cache item "/home/jeroen/.cache/rclone/vfs/storagebox-crypt{gy3BQ}/" to be canonical "/home/jeroen/.cache/rclone/vfs/storagebox-crypt{gy3BQ}"
2023/08/19 20:40:02 DEBUG : Creating backend with remote "/home/jeroen/.cache/rclone/vfsMeta/storagebox-crypt{gy3BQ}/"
2023/08/19 20:40:02 DEBUG : fs cache: renaming cache item "/home/jeroen/.cache/rclone/vfsMeta/storagebox-crypt{gy3BQ}/" to be canonical "/home/jeroen/.cache/rclone/vfsMeta/storagebox-crypt{gy3BQ}"
2023/08/19 20:40:02 INFO  : vfs cache: cleaned: objects 0 (was 0) in use 0, to upload 0, uploading 0, total size 0 (was 0)
2023/08/19 20:40:02 DEBUG : Encrypted drive 'storagebox-crypt{gy3BQ}:': Mounting on "crypt-mount/"
2

welcome to the forum,

that is what i would do.
would be curious if there is a better solution or any other solution at all.

fwiw, this is the rclone source code
https://github.com/rclone/rclone/blob/156c372cd760460f419708c75506434c48decfc8/fs/newfs.go#L43

3

I worked around this problem by generating a config file on-the-fly on a ramfs. When rclone is using a config file, it doesn't add a suffix to the cache dir name, even though the obscured password is different everytime.

Still, it feels a bit wrong to have to put passwords in a 'file' instead of using a more ephemeral mechanism like environment variables, which seem more suited for passing secret information.

I can try to make an PR to hash the config after un-obscuring the passwords. But I will only do that if more people think this is a good idea in principle.

4

Why do you insist on storing xxxxxxxxxxxxx instead of rclone obscure xxxxxxxxxxxxx? It is not needed for anything.

If for whatever reason one day you need xxxxxxxxxxxxx you can decode it (it is not really encrypted):

https://forum.rclone.org/t/how-to-retrieve-a-crypt-password-from-a-config-file/

5

I usually store only passwords, not information derived from passwords. I could consider the obscured password to be the password, but I would have to be absolutely sure the current obscuring algorithm will remain supported forever. Or I would have to store both the obscured and non-obscured password, but I prefer to have a single source of truth.

6

IMHO nobody will give you 100% guarantee that things will stay the way you like.

One solution would be to fork rclone repo and maintain your custom version.

7

nice workaround!
fwiw, with my backup script, i create on-the-fly remotes, do not use config file at all.

8

That would be my ideal too. How do you pass your password to rclone?

9

if there is no config, then there is no config password.

10

Sorry, I meant an encryption password.

11 
export RCLONE_CONFIG_STORAGEBOX-CRYPT_TYPE=crypt
export RCLONE_CONFIG_STORAGEBOX-CRYPT_REMOTE=storagebox:probeer
export RCLONE_CONFIG_STORAGEBOX-CRYPT_PASSWORD=xxxxx
export RCLONE_CONFIG_STORAGEBOX-CRYPT_PASSWORD2=xxxxx

or could use a connection string, tho i would not recommend that.

12

env vars offers nothing over disk. Instead of access, you have scope. Data or environment. That feeling of it being wrong with ramfs is a good sign. Dont use env vars unless you have that same feeling :stuck_out_tongue:

13

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.