Crypt alias with different password/password2

What is the problem you are having with rclone?

I'm currently running into an issue, where I have multiple shortcuts from different shared drives, in a "main shared drive". When rclone copy runs, the files dissappear locally and require a re-mount to get them showing again (or a kill -SIGHUP ). I realized that probably I'm not using this correctly (should've used alias instead) and was wondering if:

There's a way to pass the encryption password/password2 to the alias OR somehow get the crypts working with it's parent password/password2

Run the command 'rclone version' and share the full output of the command.

rclone v1.59.2
- os/version: slackware 15.0+ (64 bit)
- os/kernel: 5.19.14-Unraid (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.18.6
- go/linking: static
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The rclone config contents with secrets removed.

[google_drive]
type = drive
client_id =REDACTED.apps.googleusercontent.com
client_secret = REDACTED
scope = drive
token =  REDACTED
team_drive = REDACTED

[google_drive_encrypted_vfs]
type = crypt
remote = google_drive:encrypted-data
filename_encryption = standard
directory_name_encryption = true
password = REDACTED
password2 = REDACTED

[google_drive_encrypted_vfs_2]
type = crypt
remote = google_drive_encrypted_vfs:encrypted-data-sym-1
password = REDACTED-AAAA
password2 = REDACTED-BBBB

[google_drive_encrypted_vfs_3]
type = crypt
remote = google_drive_encrypted_vfs:encrypted-data-sym-2
password =REDACTED-CCCC
password2 = REDACTED-DDDD

[test]
type = alias
remote = google_drive:encrypted-data-sym-1
password = REDACTED-AAAA
password2 = REDACTED-BBBB

Appreciate the help in advance

You point alias->crypt->remote if that's what you are trying to do as you have to order them properly.

I'm not sure why you have an alias at all though so you'd have to explain the use case a bit more.

The reason trying the alias is due to the issue I'm having whereby if I rclone move/copy anything on google_drive_encrypted_vfs_2 or google_drive_encrypted_vfs_3, it dissappears locally and requires a re-mount of kill SIGHUP to make them show up again. I thought maybe using an alias would alleviate this (and get rid of google_drive_encrypted_vfs_2 and google_drive_encrypted_vfs_3) and rely only on google_drive_encrypted_vfs

I'm attaching a photo of how the setup looks in Google Drive for better illustration.

This is the "main drive" where encrypted-data-sym-1 and encrypted-data-sym-2 are shortcuts from other shared drives.

There's no mount command in the post.
There's no debug log to take a look at the issue you are describing.

Can you please share both?
What do you mean 'copy'? You do that on the mount? You run a rclone copy on the remote?

The more you describe/share the workflow and how to recreate it, if there is an issue / bug, we can fix it.

Mount Command:

rclone mount --uid 99 --gid 100 --umask 000 --allow-other --buffer-size 256M --dir-cache-time 720h --attr-timeout 720h --drive-chunk-size 512M -vv --poll-interval 10s --cache-dir /mnt/user/cache_tmp/rclone/google_drive_encrypted_vfs_2 --drive-pacer-min-sleep 10ms --drive-pacer-burst 1000 --vfs-cache-mode full --vfs-cache-max-size 50G --vfs-cache-max-age 336h --vfs-read-ahead 1G google_drive_encrypted_vfs_2: /mnt/user/data/remote_storage_2

Mount Logs:

2022/10/14 09:18:39 DEBUG : rclone: Version "v1.59.2" starting with parameters ["rcloneorig" "--config" "/boot/config/plugins/rclone/.rclone.conf" "mount" "--uid" "99" "--gid" "100" "--umask" "000" "--allow-other" "--buffer-size" "256M" "--dir-cache-time" "720h" "--attr-timeout" "720h" "--drive-chunk-size" "512M" "-vv" "--poll-interval" "10s" "--cache-dir" "/mnt/user/cache_tmp/rclone/google_drive_encrypted_vfs_2" "--drive-pacer-min-sleep" "10ms" "--drive-pacer-burst" "1000" "--vfs-cache-mode" "full" "--vfs-cache-max-size" "50G" "--vfs-cache-max-age" "336h" "--vfs-read-ahead" "1G" "google_drive_encrypted_vfs_2:" "/mnt/user/data/remote_storage_2"]
2022/10/14 09:18:39 DEBUG : Creating backend with remote "google_drive_encrypted_vfs_2:"
2022/10/14 09:18:39 DEBUG : Using config file from "/boot/config/plugins/rclone/.rclone.conf"
2022/10/14 09:18:39 DEBUG : Creating backend with remote "google_drive:encrypted-data-sym-1"
2022/10/14 09:18:39 DEBUG : google_drive: detected overridden config - adding "{GvAJ9}" suffix to name
2022/10/14 09:18:39 DEBUG : fs cache: renaming cache item "google_drive:encrypted-data-sym-1" to be canonical "google_drive{GvAJ9}:encrypted-data-sym-1"
2022/10/14 09:18:39 DEBUG : vfs cache: root is "/mnt/user/cache_tmp/rclone/google_drive_encrypted_vfs_2"
2022/10/14 09:18:39 DEBUG : vfs cache: data root is "/mnt/user/cache_tmp/rclone/google_drive_encrypted_vfs_2/vfs/google_drive_encrypted_vfs_2"
2022/10/14 09:18:39 DEBUG : vfs cache: metadata root is "/mnt/user/cache_tmp/rclone/google_drive_encrypted_vfs_2/vfsMeta/google_drive_encrypted_vfs_2"
2022/10/14 09:18:39 DEBUG : Creating backend with remote "/mnt/user/cache_tmp/rclone/google_drive_encrypted_vfs_2/vfs/google_drive_encrypted_vfs_2/"
2022/10/14 09:18:39 DEBUG : Creating backend with remote "/mnt/user/cache_tmp/rclone/google_drive_encrypted_vfs_2/vfsMeta/google_drive_encrypted_vfs_2/"
2022/10/14 09:18:39 DEBUG : Encrypted drive 'google_drive_encrypted_vfs_2:': Mounting on "/mnt/user/data/remote_storage_2"
2022/10/14 09:18:40 DEBUG : : Root: 
2022/10/14 09:18:40 DEBUG : : >Root: node=/, err=<nil>
2022/10/14 09:18:49 DEBUG : Google drive root 'encrypted-data-sym-1': Checking for changes on remote
2022/10/14 09:18:59 DEBUG : Google drive root 'encrypted-data-sym-1': Checking for changes on remote
2022/10/14 09:19:09 DEBUG : Google drive root 'encrypted-data-sym-1': Checking for changes on remote
2022/10/14 09:19:19 DEBUG : Google drive root 'encrypted-data-sym-1': Checking for changes on remote
2022/10/14 09:19:28 DEBUG : /: Attr: 
2022/10/14 09:19:28 DEBUG : /: >Attr: attr=valid=720h0m0s ino=0 size=0 mode=drwxrwxrwx, err=<nil>
2022/10/14 09:19:29 DEBUG : Google drive root 'encrypted-data-sym-1': Checking for changes on remote
2022/10/14 09:19:37 DEBUG : /: ReadDirAll: 
2022/10/14 09:19:37 DEBUG : /: >ReadDirAll: item=5, err=<nil>
2022/10/14 09:19:37 DEBUG : /: Attr: 
2022/10/14 09:19:37 DEBUG : /: >Attr: attr=valid=720h0m0s ino=0 size=0 mode=drwxrwxrwx, err=<nil>
2022/10/14 09:19:37 DEBUG : /: Lookup: name="MyFiles"
2022/10/14 09:19:37 DEBUG : /: >Lookup: node=MyFiles/, err=<nil>
2022/10/14 09:19:37 DEBUG : MyFiles/: Attr: 
2022/10/14 09:19:37 DEBUG : MyFiles/: >Attr: attr=valid=720h0m0s ino=0 size=0 mode=drwxrwxrwx, err=<nil>
2022/10/14 09:19:37 DEBUG : /: Lookup: name="mountcheck"
2022/10/14 09:19:37 DEBUG : /: >Lookup: node=mountcheck, err=<nil>
2022/10/14 09:19:37 DEBUG : mountcheck: Attr: 
2022/10/14 09:19:37 DEBUG : mountcheck: >Attr: a=valid=720h0m0s ino=0 size=0 mode=-rw-rw-rw-, err=<nil>
2022/10/14 09:19:37 DEBUG : /: Lookup: name="test_5.txt"
2022/10/14 09:19:37 DEBUG : /: >Lookup: node=test_5.txt, err=<nil>
2022/10/14 09:19:37 DEBUG : test_5.txt: Attr: 
2022/10/14 09:19:37 DEBUG : test_5.txt: >Attr: a=valid=720h0m0s ino=0 size=0 mode=-rw-rw-rw-, err=<nil>
2022/10/14 09:19:37 DEBUG : /: Attr: 
2022/10/14 09:19:37 DEBUG : /: >Attr: attr=valid=720h0m0s ino=0 size=0 mode=drwxrwxrwx, err=<nil>
2022/10/14 09:19:39 DEBUG : Google drive root 'encrypted-data-sym-1': Checking for changes on remote
2022/10/14 09:19:40 DEBUG : vfs cache RemoveNotInUse (maxAge=1209600000000000, emptyOnly=false): item MyFiles/test.mp4 not removed, freed 0 bytes
2022/10/14 09:19:40 DEBUG : vfs cache RemoveNotInUse (maxAge=1209600000000000, emptyOnly=false): item MyFiles/test2.mp4 not removed, freed 0 bytes
2022/10/14 09:19:40 DEBUG : vfs cache RemoveNotInUse (maxAge=1209600000000000, emptyOnly=false): item MyFiles/test3.mp4 not removed, freed 0 bytes
2022/10/14 09:19:40 DEBUG : vfs cache RemoveNotInUse (maxAge=1209600000000000, emptyOnly=false): item MyFiles/test4.mp4 not removed, freed 0 bytes
2022/10/14 09:19:40 DEBUG : vfs cache RemoveNotInUse (maxAge=1209600000000000, emptyOnly=false): item MyFiles/test5.mp4 not removed, freed 0 bytes
2022/10/14 09:19:40 INFO  : vfs cache: cleaned: objects 5 (was 5) in use 0, to upload 0, uploading 0, total size 2.041Gi (was 2.041Gi)
2022/10/14 09:19:49 DEBUG : Google drive root 'encrypted-data-sym-1': Checking for changes on remote
2022/10/14 09:19:59 DEBUG : Google drive root 'encrypted-data-sym-1': Checking for changes on remote
2022/10/14 09:20:03 DEBUG : /: ReadDirAll: 
2022/10/14 09:20:03 DEBUG : /: >ReadDirAll: item=5, err=<nil>
2022/10/14 09:20:03 DEBUG : /: Attr: 
2022/10/14 09:20:03 DEBUG : /: >Attr: attr=valid=720h0m0s ino=0 size=0 mode=drwxrwxrwx, err=<nil>
2022/10/14 09:20:03 DEBUG : /: Attr: 
2022/10/14 09:20:03 DEBUG : /: >Attr: attr=valid=720h0m0s ino=0 size=0 mode=drwxrwxrwx, err=<nil>
2022/10/14 09:20:08 DEBUG : /: ReadDirAll: 
2022/10/14 09:20:08 DEBUG : /: >ReadDirAll: item=5, err=<nil>
2022/10/14 09:20:08 DEBUG : /: Attr: 
2022/10/14 09:20:08 DEBUG : /: >Attr: attr=valid=720h0m0s ino=0 size=0 mode=drwxrwxrwx, err=<nil>
2022/10/14 09:20:08 DEBUG : /: Attr: 
2022/10/14 09:20:08 DEBUG : /: >Attr: attr=valid=720h0m0s ino=0 size=0 mode=drwxrwxrwx, err=<nil>
2022/10/14 09:20:09 DEBUG : Google drive root 'encrypted-data-sym-1': Checking for changes on remote
2022/10/14 09:20:19 DEBUG : Google drive root 'encrypted-data-sym-1': Checking for changes on remote

rclone move command

rclone --config /boot/config/plugins/rclone/.rclone.conf move /mnt/user/data/staged_storage_2 google_drive_encrypted_vfs_2: --drive-service-account-file=/mnt/user/appdata/other/rclone/service_accounts/sa-account-9.json --user-agent=google_drive_encrypted_vfs_2 -vv --buffer-size 512M --drive-chunk-size 512M --tpslimit 8 --checkers 8 --transfers 4 --order-by modtime,ascending --exclude downloads/** --exclude tmp/** --exclude *fuse_hidden* --exclude *_HIDDEN --exclude .recycle** --exclude .Recycle.Bin/** --exclude *.backup~* --exclude *.partial~* --drive-stop-on-upload-limit --delete-empty-src-dirs

rclone move logs

2022/10/14 09:18:47 INFO  : Starting transaction limiter: max 8 transactions/s with burst 1
2022/10/14 09:18:47 DEBUG : rclone: Version "v1.59.2" starting with parameters ["rcloneorig" "--config" "/boot/config/plugins/rclone/.rclone.conf" "--config" "/boot/config/plugins/rclone/.rclone.conf" "move" "/mnt/user/data/staged_storage_2" "google_drive_encrypted_vfs_2:" "--drive-service-account-file=/mnt/user/appdata/other/rclone/service_accounts/sa-account-9.json" "--user-agent=google_drive_encrypted_vfs_2" "-vv" "--buffer-size" "512M" "--drive-chunk-size" "512M" "--tpslimit" "8" "--checkers" "8" "--transfers" "4" "--order-by" "modtime,ascending" "--exclude" "downloads/**" "--exclude" "tmp/**" "--exclude" "*fuse_hidden*" "--exclude" "*_HIDDEN" "--exclude" ".recycle**" "--exclude" ".Recycle.Bin/**" "--exclude" "*.backup~*" "--exclude" "*.partial~*" "--drive-stop-on-upload-limit" "--delete-empty-src-dirs"]
2022/10/14 09:18:47 DEBUG : Creating backend with remote "/mnt/user/data/staged_storage_2"
2022/10/14 09:18:47 DEBUG : Using config file from "/boot/config/plugins/rclone/.rclone.conf"
2022/10/14 09:18:47 DEBUG : Creating backend with remote "google_drive_encrypted_vfs_2:"
2022/10/14 09:18:47 DEBUG : Creating backend with remote "google_drive:encrypted-data-sym-1"
2022/10/14 09:18:47 DEBUG : google_drive: detected overridden config - adding "{xZ0x2}" suffix to name
2022/10/14 09:18:48 DEBUG : fs cache: renaming cache item "google_drive:encrypted-data-sym-1" to be canonical "google_drive{xZ0x2}:encrypted-data-sym-1"
2022/10/14 09:18:48 DEBUG : Encrypted drive 'google_drive_encrypted_vfs_2:': Waiting for checks to finish
2022/10/14 09:18:48 DEBUG : Encrypted drive 'google_drive_encrypted_vfs_2:': Waiting for transfers to finish
2022/10/14 09:18:51 DEBUG : test_5.txt: md5 = 644594527b4647799e3ca44c457588a5 OK
2022/10/14 09:18:51 INFO  : test_5.txt: Copied (new)
2022/10/14 09:18:51 INFO  : test_5.txt: Deleted
2022/10/14 09:18:51 INFO  : 
Transferred:             32 B / 32 B, 100%, 15 B/s, ETA 0s
Checks:                 2 / 2, 100%
Deleted:                1 (files), 0 (dirs)
Renamed:                1
Transferred:            1 / 1, 100%
Elapsed time:         3.7s

2022/10/14 09:18:51 DEBUG : 8 go routines active

The workflow is as follows

  • Files are placed in /mnt/user/data/staged_storage_2
  • On a timer (and if not already uploading) files are rclone moved from /mnt/user/data/staged_storage_2 to google_drive_encrypted_vfs_2:
  • These files are supposed to appear in /mnt/user/data/remote_storage_2 but instead they dissappear.
  • The files can be seen on Google Drive (and encrypted) after the rclone move, however, I have to either remount or kill -SIGHUP for them to show up in /mnt/user/data/remote_storage_2

Please let me know if you require any additional information

You can make your alias work like this with different passwords. google_drive_encrypted needs to be a crypt remote pointing at the google drive. You can override things in its config by passing them as shown below.

[test]
type = alias
remote = google_drive_encrypted,password="REDACTED-AAAA",password2="REDACTED-BBBB":encrypted-data-sym-1

Do you think this would work better than my current approach? (i.e. having another crypt pointing to a shortcut within another shared drive?)

I'd say use which ever approach makes the most sense to you!

Unfortunately, the reason of trying out aliases is due to the problems I've been having with dissappearing files. From what it sounds like, it seems that using aliases won't alleviate this. I've added some more details in my previous post (Crypt alias with different password/password2 - #5 by teh0wner)

@ncw Neat, I didn't know and would like to learn. Can you point me to the docs or code?

1 Like

I would first try removing this from your mount command (the default is 1s):

More information here:
https://rclone.org/commands/rclone_mount/#attribute-caching

Thanks. Actually that's a flag I was playing around with to see if it makes any changes. I didn't use to have it, and the problem would still exhibit itself.

Next I would try also removing this from your mount command (the default is 5m):

and then wait minimum 5 minutes after the move has completed.

More info here:
https://rclone.org/commands/rclone_mount/#vfs-directory-cache

@Ole this is a remote defined with a connection string which you can use to override some parameters in an existing remote.

It would probably be a good idea to document that in the alias backend too as people often want to override things in the alias remote definition and try to just write the parameters in the config (as above) rather than the connection string.

I suppose the alias backend could fairly easily supply any extra parameters as parameters to the remote...

So currently this does not work

[myalias]
type = alias
remote = potato:
param1 = one
param2 = two

Whereas this does work

[myalias]
type = alias
remote = potato,param1="one",param2="two":

It would be possible to make the first work also fairly easily???

1 Like

Thanks, I agree a hint and a link in documentation of the relevant backends would make the interested aware of the possibility and the correct way to do it. I guess I just read the current --alias-remote section to literally, like "Can only be ..."

I will make a GitHub issue to discuss wording etc. outside this thread.

I would refrain from this to keep things simple, both code and documentation.

I think it is good that people get to read the section on connection strings, including the caveats and limitations

hi,
if filters are complex, then all the more so, connection strings

whereas, this is much more approachable.

[myalias]
type = alias
remote = potato:
param1 = one
param2 = two

what if the wrapped remote, potato: is

[potato]
param1 = notone

does myalias: values override potato:?

Yes the myalias values would override potato.

In the implementation I have in mind, what rclone would do is build a connection string for you using the alias parameters so the two methods of setting parameters would be identical.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.