Rclone mount creates different VFS cache dirs for different obscured permutations of the same password if config is overridden

Jaitsu · June 16, 2024, 11:27am

What is the problem you are having with rclone?

I have a somewhat unusual use case - using rclone to encrypt a folder on a flash drive, along with a copy of Rclone and a batch file to obscure and pass it the password and access the folder.

Each time the same encrypted folder is mounted using rclone mount using a different obscured form of the same password, with --vfs-cache-mode full, rclone creates a new VFS cache directory with a new random string, e.g. crypt{abcde}/crypt{fghij}, and doesn't detect/maintain/clean up old versions.

If the mount is not left running long enough for everything to reach --vfs-cache-max-age, this potentially leads to large amounts of junk accumulating in the vfscache folder.

Looking at the source, it looks like this is caused by an MD5 being run on the overridden config - and because obscure isn't deterministic, it's technically different even though the revealed password is still effectively the same, yielding a different hash and thus a new suffix.

Run the command 'rclone version' and share the full output of the command.

- os/version: Microsoft Windows 11 Pro 23H2 (64 bit)
- os/kernel: 10.0.22631.3737 (x86_64)
- os/type: windows
- os/arch: amd64
- go/version: go1.22.4
- go/linking: static
- go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

N/A, I'm using Rclone to encrypt a local directory on a flash drive.

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

start %CD:~0,2%\Rclone\Windows\rclone.exe mount crypt: R: --vfs-cache-mode full --volname Crypt --rc --rc-no-auth -vv

The rclone config contents with secrets removed.

N/A as I am using entirely environment variables set in my batch file, apart from a blank rclone.conf to silence the warning about the file being missing. Relevant environment variables:

set RCLONE_CONFIG_CRYPT_TYPE=crypt
set RCLONE_CONFIG_CRYPT_FILENAME_ENCRYPTION=standard
set RCLONE_CONFIG_CRYPT_DIRECTORY_NAME_ENCRYPTION=true
set RCLONE_CONFIG_CRYPT_REMOTE=%CD:~0,2%\Vault
FOR /F "delims=" %%i IN ('%CD:~0,2%\Rclone\Windows\rclone.exe obscure "!rc_enc!"') DO set "RCLONE_CONFIG_CRYPT_PASSWORD=%%i"

I am using a batch file to prompt for the password to mount and access the folder, rather than storing the password anywhere, hence re-obscuring the password for each run.

Note that my batch file doesn't have anything to do with the behavior - the same thing happens when just running the command with different obscured forms of the password.

A log from the command with the `-vv` flag

2024/06/16 06:54:49 DEBUG : rclone: Version "v1.67.0" starting with parameters ["E:\\Rclone\\Windows\\rclone.exe" "mount" "crypt:" "R:" "--vfs-cache-mode" "full" "--volname" "Crypt" "--rc" "--rc-no-auth" "-vv"]
2024/06/16 06:54:49 NOTICE: Serving remote control on http://127.0.0.1:5572/
2024/06/16 06:54:49 DEBUG : Creating backend with remote "crypt:"
2024/06/16 06:54:49 DEBUG : Using config file from "E:\\Rclone\\Windows\\rclone.conf"
2024/06/16 06:54:49 DEBUG : Setting type="crypt" for "crypt" from environment variable RCLONE_CONFIG_CRYPT_TYPE
2024/06/16 06:54:49 DEBUG : Setting remote="E:\\Vault" for "crypt" from environment variable RCLONE_CONFIG_CRYPT_REMOTE
2024/06/16 06:54:49 DEBUG : Setting filename_encryption="standard" for "crypt" from environment variable RCLONE_CONFIG_CRYPT_FILENAME_ENCRYPTION
2024/06/16 06:54:49 DEBUG : Setting directory_name_encryption="true" for "crypt" from environment variable RCLONE_CONFIG_CRYPT_DIRECTORY_NAME_ENCRYPTION
2024/06/16 06:54:49 DEBUG : Setting password="[REDACTED]" for "crypt" from environment variable RCLONE_CONFIG_CRYPT_PASSWORD
2024/06/16 06:54:49 DEBUG : crypt: detected overridden config - adding "{cRbgf}" suffix to name
2024/06/16 06:54:49 DEBUG : Setting remote="E:\\Vault" for "crypt" from environment variable RCLONE_CONFIG_CRYPT_REMOTE
2024/06/16 06:54:49 DEBUG : Setting filename_encryption="standard" for "crypt" from environment variable RCLONE_CONFIG_CRYPT_FILENAME_ENCRYPTION
2024/06/16 06:54:49 DEBUG : Setting directory_name_encryption="true" for "crypt" from environment variable RCLONE_CONFIG_CRYPT_DIRECTORY_NAME_ENCRYPTION
2024/06/16 06:54:49 DEBUG : Setting password="[REDACTED]" for "crypt" from environment variable RCLONE_CONFIG_CRYPT_PASSWORD
2024/06/16 06:54:49 DEBUG : Creating backend with remote "E:\\Vault"
2024/06/16 06:54:49 DEBUG : fs cache: renaming cache item "E:\\Vault" to be canonical "//?/E:/Vault"
2024/06/16 06:54:49 DEBUG : fs cache: renaming cache item "crypt:" to be canonical "crypt{cRbgf}:"
2024/06/16 06:54:49 INFO  : Encrypted drive 'crypt{cRbgf}:': poll-interval is not supported by this remote
2024/06/16 06:54:49 DEBUG : vfs cache: root is "C:\\Users\\Jaitsu\\AppData\\Local\\rclone"
2024/06/16 06:54:49 DEBUG : vfs cache: data root is "\\\\?\\C:\\Users\\Jaitsu\\AppData\\Local\\rclone\\vfs\\crypt{cRbgf}"
2024/06/16 06:54:49 DEBUG : vfs cache: metadata root is "\\\\?\\C:\\Users\\Jaitsu\\AppData\\Local\\rclone\\vfsMeta\\crypt{cRbgf}"
2024/06/16 06:54:49 DEBUG : Creating backend with remote "C:/Users/Jaitsu/AppData/Local/rclone/vfs/crypt{cRbgf}/"
2024/06/16 06:54:49 DEBUG : fs cache: renaming cache item "C:/Users/Jaitsu/AppData/Local/rclone/vfs/crypt{cRbgf}/" to be canonical "//?/C:/Users/Jaitsu/AppData/Local/rclone/vfs/crypt{cRbgf}"
2024/06/16 06:54:49 DEBUG : Creating backend with remote "C:/Users/Jaitsu/AppData/Local/rclone/vfsMeta/crypt{cRbgf}/"
2024/06/16 06:54:49 DEBUG : fs cache: renaming cache item "C:/Users/Jaitsu/AppData/Local/rclone/vfsMeta/crypt{cRbgf}/" to be canonical "//?/C:/Users/Jaitsu/AppData/Local/rclone/vfsMeta/crypt{cRbgf}"
2024/06/16 06:54:49 DEBUG : Network mode mounting is disabled
2024/06/16 06:54:49 DEBUG : Mounting on "R:" ("Crypt")
2024/06/16 06:54:49 DEBUG : Encrypted drive 'crypt{cRbgf}:': Mounting with options: ["-o" "attr_timeout=1" "-o" "uid=-1" "-o" "gid=-1" "--FileSystemName=rclone" "-o" "volname=Crypt"]
2024/06/16 06:54:49 INFO  : vfs cache: cleaned: objects 0 (was 0) in use 0, to upload 0, uploading 0, total size 0 (was 0)
2024/06/16 06:54:49 DEBUG : Encrypted drive 'crypt{cRbgf}:': Init:
2024/06/16 06:54:49 DEBUG : Encrypted drive 'crypt{cRbgf}:': >Init:
2024/06/16 06:54:49 DEBUG : /: Statfs:
2024/06/16 06:54:49 DEBUG : /: >Statfs: stat={Bsize:4096 Frsize:4096 Blocks:15620864 Bfree:9654848 Bavail:9654848 Files:1000000000 Ffree:1000000000 Favail:0 Fsid:0 Flag:0 Namemax:255}, errc=0
2024/06/16 06:54:49 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2024/06/16 06:54:49 DEBUG : /: >Getattr: errc=0
2024/06/16 06:54:49 DEBUG : /: Readlink:
2024/06/16 06:54:49 DEBUG : /: >Readlink: linkPath="", errc=-40
2024/06/16 06:54:49 DEBUG : /: Getxattr: name="non-existant-a11ec902d22f4ec49003af15282d3b00"
2024/06/16 06:54:49 DEBUG : /: >Getxattr: errc=-40, value=""
The service rclone has been started.
2024/06/16 06:54:49 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2024/06/16 06:54:49 DEBUG : /: >Getattr: errc=0
2024/06/16 06:54:49 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2024/06/16 06:54:49 DEBUG : /: >Getattr: errc=0
2024/06/16 06:54:49 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2024/06/16 06:54:49 DEBUG : /: >Getattr: errc=0
2024/06/16 06:54:49 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2024/06/16 06:54:49 DEBUG : /: >Getattr: errc=0
2024/06/16 06:54:49 DEBUG : /: Opendir:
2024/06/16 06:54:49 DEBUG : /: Opendir:
2024/06/16 06:54:49 DEBUG : /: OpenFile: flags=O_RDONLY, perm=-rwxrwxrwx
2024/06/16 06:54:49 DEBUG : /: >OpenFile: fd=/ (r), err=<nil>
2024/06/16 06:54:49 DEBUG : /: >Opendir: errc=0, fh=0x0
2024/06/16 06:54:49 DEBUG : /: OpenFile: flags=O_RDONLY, perm=-rwxrwxrwx
2024/06/16 06:54:49 DEBUG : /: >OpenFile: fd=/ (r), err=<nil>
2024/06/16 06:54:49 DEBUG : /: >Opendir: errc=0, fh=0x1
2024/06/16 06:54:49 DEBUG : /: Statfs:
2024/06/16 06:54:49 DEBUG : /: Releasedir: fh=0x1
2024/06/16 06:54:49 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2024/06/16 06:54:49 DEBUG : /: >Getattr: errc=0
2024/06/16 06:54:49 DEBUG : /: >Releasedir: errc=0
2024/06/16 06:54:49 DEBUG : /: >Statfs: stat={Bsize:4096 Frsize:4096 Blocks:15620864 Bfree:9654848 Bavail:9654848 Files:1000000000 Ffree:1000000000 Favail:0 Fsid:0 Flag:0 Namemax:255}, errc=0
2024/06/16 06:54:49 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2024/06/16 06:54:49 DEBUG : /: Releasedir: fh=0x0
2024/06/16 06:54:49 DEBUG : /: >Releasedir: errc=0
2024/06/16 06:54:49 DEBUG : /: >Getattr: errc=0
2024/06/16 06:54:49 DEBUG : /: Opendir:
2024/06/16 06:54:49 DEBUG : /: OpenFile: flags=O_RDONLY, perm=-rwxrwxrwx
2024/06/16 06:54:49 DEBUG : /: >OpenFile: fd=/ (r), err=<nil>
2024/06/16 06:54:49 DEBUG : /: >Opendir: errc=0, fh=0x0
2024/06/16 06:54:49 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2024/06/16 06:54:49 DEBUG : /: Releasedir: fh=0x0
2024/06/16 06:54:49 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2024/06/16 06:54:49 DEBUG : /: >Getattr: errc=0
2024/06/16 06:54:49 DEBUG : /: >Getattr: errc=0
2024/06/16 06:54:49 DEBUG : /: >Releasedir: errc=0
2024/06/16 06:54:49 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2024/06/16 06:54:49 DEBUG : /: >Getattr: errc=0
2024/06/16 06:54:49 DEBUG : /: Opendir:
2024/06/16 06:54:49 DEBUG : /: OpenFile: flags=O_RDONLY, perm=-rwxrwxrwx
2024/06/16 06:54:49 DEBUG : /: >OpenFile: fd=/ (r), err=<nil>
2024/06/16 06:54:49 DEBUG : /: >Opendir: errc=0, fh=0x0
2024/06/16 06:54:49 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2024/06/16 06:54:49 DEBUG : /: >Getattr: errc=0
2024/06/16 06:54:49 DEBUG : /: Releasedir: fh=0x0
2024/06/16 06:54:49 DEBUG : /: >Releasedir: errc=0
2024/06/16 06:54:49 DEBUG : /: Opendir:
2024/06/16 06:54:49 DEBUG : /: OpenFile: flags=O_RDONLY, perm=-rwxrwxrwx
2024/06/16 06:54:49 DEBUG : /: >OpenFile: fd=/ (r), err=<nil>
2024/06/16 06:54:49 DEBUG : /: >Opendir: errc=0, fh=0x0
2024/06/16 06:54:49 DEBUG : /: Releasedir: fh=0x0
2024/06/16 06:54:49 DEBUG : /: >Releasedir: errc=0

asdffdsa · June 16, 2024, 1:41pm

hi,
that is a feature, not a bug. has been discussed a number of times in the forum
Multiple --cache-dir subdirs for the same remote
Detected overridden config
and github
https://github.com/rclone/rclone/issues/7364

just curious, what is the practical difference from using an encrypted config with password versus your approach?
and if security is important, should use two password for crypt remote.

Jaitsu · June 16, 2024, 2:20pm

Ah, honestly, didn't even realize config encryption was an option when I set out to come up with my own way to pass it the password. That seems like it should work, I'll try that out later.

Even if that does work, though, I do still think that the password in particular should be being decoded/"revealed" before the hash is performed where I linked, given the purpose of the suffixes, since it's the same password/decrypts the same data and yields an otherwise identical connection where there's no reason to break the VFS cache association.

system · July 16, 2024, 2:20pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.