Passing mount options (like vfs-cache-mode) when using "rclone rc mount/mount"

What is the problem you are having with rclone?

I would like to run a single instance of "rclone rcd", manage multiple mounts with "rclone rc mount/mount" and still provide parameters like --vfs-cache-mode for the individual mounts. I've tried both the command line and JSON-object way, but I can't get rclone to recognize the extended mount options when running it this way.

The reason is I'm running a handful of mounts and would prefer central control through a single port, instead of running one instance each.

Any help is appreciated. :slight_smile:

What is your rclone version (output from rclone version)

(note: latest rclone git version installed on Gentoo due to testing the vfs-cache-mode full changes. Current checkout v1.52.2-212-g80d2f381)

rclone v1.52.2-DEV

  • os/arch: linux/amd64
  • go version: go1.14.2

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Gentoo Linux, 64 bit

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone rcd --rc-user=USER --rc-pass=PASS
rclone rc mount/mount --rc-user=USER --rc-pass=PASS fs=drive_YYYY: mountPoint=/path/to/mountpoint/YYYY -a allow-other -o vfs-cache-mode=full -o cache-dir=/path/to/cachedir/ -o vfs-cache-poll-interval=20m -o vfs-cache-max-age=8760h -o attr-timeout=8700h -o dir-cache-time=8760h -o poll-interval=30s

and alternatively as JSON:

rclone rc mount/mount --rc-user=USER --rc-pass=PASS --json '{ "fs":"drive_YYYY:", "mountPoint":"/path/to/mountpoint/YYYY", "vfs-cache-mode":"full", "cache-dir":"/path/to/cachedir/", "vfs-cache-poll-interval":"20m", "vfs-cache-max-age":"8760h", "attr-timeout":"8700h", "dir-cache-time":"8760h", "poll-interval":"30s" }'

The rclone config contents with secrets removed.

[drive_YYYY]
type = drive
scope = drive
token = ...
client_id = ...
client_secret = ...
team_drive = ...
server_side_across_configs = true

A log from the command with the -vv flag

Commandline options version:

2020/07/14 03:09:46 DEBUG : rc: "mount/mount": with parameters map[arg:[allow-other] fs:drive_YYYY: mountPoint:/path/to/mountpoint/YYYY opt:map[attr-timeout:8700h cache-dir:/path/to/cachedir dir-cache-time:8760h poll-interval:30s vfs-cache-max-age:8760h vfs-cache-mode:full vfs-cache-poll-interval:20m]]
2020/07/14 03:09:46 DEBUG : Google drive root '': Mounting on "/path/to/mountpoint/YYYY"
2020/07/14 03:09:46 DEBUG : Google drive root '': Re-using VFS from active cache
2020/07/14 03:09:46 DEBUG : Mount for Google drive root '' created at /path/to/mountpoint/YYYY using mount
2020/07/14 03:09:46 DEBUG : rc: "mount/mount": reply map[]: <nil>
2020/07/14 03:09:46 DEBUG : : Root: 
2020/07/14 03:09:46 DEBUG : : >Root: node=/, err=<nil>

JSON object version:

2020/07/14 03:06:58 DEBUG : rc: "mount/mount": with parameters map[attr-timeout:8700h dir-cache-time:8760h fs:drive_YYYY: mountPoint:/path/to/mountpoint/YYYY poll-interval:30s vfs:map[cache-dir:/path/to/cachedir vfs-cache-max-age:8760h vfs-cache-mode:full vfs-cache-poll-interval:20m]]
2020/07/14 03:06:58 DEBUG : Google drive root '': Mounting on "/path/to/mountpoint/YYYY"
2020/07/14 03:06:58 DEBUG : Google drive root '': Re-using VFS from active cache
2020/07/14 03:06:58 DEBUG : Mount for Google drive root '' created at /path/to/mountpoint/YYYY using mount
2020/07/14 03:06:58 DEBUG : rc: "mount/mount": reply map[]: <nil>
2020/07/14 03:06:58 DEBUG : : Root: 
2020/07/14 03:06:58 DEBUG : : >Root: node=/, err=<nil>

hello and welcome to the forum,

you can try this
https://rclone.org/docs/#config-file

for example

export  RCLONE_CONFIG_FTP1_TYPE=ftp
export  RCLONE_CONFIG_FTP1_HOST=127.0.0.1
export  RCLONE_CONFIG_FTP1_USER=user01
export  RCLONE_CONFIG_FTP1_PASS=`$rcmd obscure password`
rclone lsd FTP1:

Hey,

Thank you for your suggestion. I've finally had some time to try passing arguments as environment variables like in your example, but they won't take like that either.

I've tested with

RCLONE_CONFIG_REMOTENAME_CACHE_DIR=/path/to/cachedir
RCLONE_CONFIG_REMOTENAME_VFS_CACHE_MODE=full

and then mounting the remote with bare minimum options. Doesn't seem to work, since the cache directory is not created. If I pass both parameters to mount on the command line it creates a new cache directory instantly.

Are there any other ways of defining options for a remote that don't normally go into the rclone.config? What am I doing wrong here? :slight_smile:

sorry, i realize now my suggestion will not work.

the RCLONE_CONFIG_ only works for backends.
mount is a command, not a backend

No problem. :slight_smile:
I thought as much, but tried it anyway because maybe it was an undocumented feature.

If anyone has any idea how to pass mount options using rc mount/mount that'd be great.

there are a bunch of -o options that you are passing.
are none of them working?

perhaps try
https://rclone.org/rc/#options-set

Can't really say if any of them are working, haven't tested that far. It's harder to test intricacies of caching than to binarily check if it's even working or not (=directory is being created). As long as cache-dir and vs-cache-mode don't have any effect, the whole exercise is moot.

options/set looks interesting, but I doubt rclone can change the cache-dir or mode mid-operation. Also I don't know exactly how to options/set for a single mount. Do I do it before on the remote (weird, since these are mount related options), or do I need to change them after mount (also problematic, since I assume some core parameters can not be changed. Also what happens if I mount my "ever-cache" that expires after a year with default options and rclone starts cleaning up?).

Maybe I should have chosen a simpler example. Let's change my main goal to:
"rc mount/mount", while providing cache-dir and vfs-cache-mode per mount on the command line.
(I'd also be happy if the mounts need to be preconfigured somewhere, my main goal is to run all mounts, no matter how complex, under single "rclone rcd" instance)

If I can get those two options to take then there should be a way to get the rest working. :slight_smile:

You can pass any options in options/set in the VFS flags section

However the mount really needs the VFS flags and there isn't a way of passing those at the moment.

I've had a go at this here

v1.52.2-235-g26138a8b-fix-vfs-rc-beta on branch fix-vfs-rc (uploaded in 15-30 mins)

mount/mount takes the following parameters

  • fs - a remote path to be mounted (required)
  • mountPoint: valid path on the local machine (required)
  • mountType: One of the values (mount, cmount, mount2) specifies the mount implementation to use
  • vfsOpt: a JSON object with VFS options in.

Eg

rclone rc mount/mount fs=mydrive: mountPoint=/home/<user>/mountPoint
rclone rc mount/mount fs=mydrive: mountPoint=/home/<user>/mountPoint mountType=mount
rclone rc mount/mount fs=TestDrive: mountPoint=/mnt/tmp vfsOpt='{"CacheMode": 2}'

The vfsOpt are as described in options/get and can be seen in the the
"vfs" section when running:

rclone rc options/get

Annoyingly the cache modes need to be set as integers 0=off, 1=minimal, 2=writes, 3=full - not sure how to fix this!

Amazing! Thanks for the quick fix. I've checked out the branch and it seems to work. "seems" because I currently cannot verify if it accepted all parameters. But from the top:

rclone rcd --rc-user=USER --rc-pass=PASS -v -v --cache-dir /path/to/testcache

and then

rclone rc mount/mount --rc-user=USER --rc-pass=PASS fs=REMOTE: mountPoint="/path/to/mount" vfsOpt='{"CacheMode": 3, "CachePollInterval": 1200000000000, "CacheMaxAge": 31536000000000000, "DirCacheTime": 31536000000000000, "PollInterval": 30000000000}' -a allow-other -o attr-timeout=8700h

And I see a new cache directory created and I can verify vfs-cache-mode=full (new chunked) caching. This gets me 90% of the way, but three minor issues remain. :slight_smile:

  1. I can't confirm whether the all options are recognized. Running rclone rc options/get only gives me the default values, even when passing fs=REMOTE:
  2. I haven't found a way to set --allow-other and --attr-timeout. In case of --allow-other passing them on the commandline does not get forwarded to the mount command. Tried wrapping them in json, but you can't use parameters and --json in the same command.
  3. Minor issue: As you can see you need to pass time in nanoseconds instead of being able to specify hours. While i usually appreciate fine granularity, I think this might be a bit over the top. :smiley: But that's only really a "problem" once and does not impact functionality.

Again, amazingly fast response with an on-point fix for my problem.

Now all I need is to be able to somehow maybe set attr-timeout and enable allow-other, and be able to verify the settings (cache is currently 280GB real / 34TB virtual in 17k files. It would suck for that to get purged and requiring a rebuild...).

for some reason, the windows files have not been uploaded?

Small update:
Checked on a testcache with a 60s CacheMaxAge and 90s CachePollInterval and it purged cache as expected, while it didn't when using the high values from above. So the parameters seem to work. Still would be nice to be able to check though. :slight_smile:

I can work around allow_other.

Not sure how much impact an attr-timeout > 1s would have with a large number of directories and files. If it's negligible then I don't really need it. Still would be nice to be able to set that somehow. :slight_smile:

Another update:
Yeah, now I remember why I needed allow-other. Because without it I cannot access the mounted folders through samba shares. :man_facepalming:

Any idea on how to activate that flag?

This needs a bit of extra VFS API to fix

I guess it could be mount API too.

Hmm, those are mount options rather than VFS options so they will need a mountOpt parameter which I'll have to implement.

They should probably appear in the options blocks too.

Yes this is slightly annoying! I could probably fix it by implementing JSON unmarshalling for the types so they could take a string "15s" or an integer. That wouldn't be very hard and would fix the VFS cache modes too...

:smiley: I've been thinking about doing this I just need an excuse!

I'll produce an update tomorrow which can set the mount flags

Looks like the Windows build failed - I'll investigate tomorrow!

1 Like

OK here is the next version

v1.52.2-239-g7d78817d-fix-vfs-rc-beta on branch fix-vfs-rc (uploaded in 15-30 mins)

This has mountOpt and vfsOpt support. THis has also made the mount options globally settable. You can read the mount options using mount/listmounts too.

This takes the following parameters

  • fs - a remote path to be mounted (required)
  • mountPoint: valid path on the local machine (required)
  • mountType: One of the values (mount, cmount, mount2) specifies the mount implementation to use
  • mountOpt: a JSON object with Mount options in.
  • vfsOpt: a JSON object with VFS options in.

Eg

rclone rc mount/mount fs=mydrive: mountPoint=/home/<user>/mountPoint
rclone rc mount/mount fs=mydrive: mountPoint=/home/<user>/mountPoint mountType=mount
rclone rc mount/mount fs=TestDrive: mountPoint=/mnt/tmp vfsOpt='{"CacheMode": 2}' mountOpt='{"AllowOther": true}'

The vfsOpt are as described in options/get and can be seen in the the
"vfs" section when running and the mountOpt can be seen in the "mount" section.

rclone rc options/get
2 Likes

thanks,

i do not see the windows files?

What can I say, it just works. :smiley:

Starting the server:

rclone rcd --rc-user=USER --rc-pass=PASS -v -v --cache-dir /path/to/testcache

Setting minimum global defaults:

rclone rc options/set --rc-user=USER --rc-pass=PASS --json '{"vfs": {"CacheMode": 3}, "mount": {"AllowOther": true}}'

and finally mounting a test drive

rclone rc mount/mount --rc-user=USER --rc-pass=PASS fs=REMOTE: mountPoint="/path/to/mount"

And everything behaves like I would expect.

I changed my mount script, put in all my standard options in the new format, and after that checking mount options with

rclone rc mount/listmounts --rc-user=USER --rc-pass=PASS

gives sane results. So everything is working proper now. :slight_smile:

Thanks for the quick help. I'll be staying on this branch till the changes get merged into master. Finally a clean setup without a different port per mount instance. :smiley:

Thanks for testing :smiley:

I'll work out why the Windows build failed again!

I've fixed the Windows build and merged this to master now so it should be in the latest beta in 15 mins or so!

1 Like

thanks for the windows beta.
i am trying to test the beta, make sure it all works on windows.

getting an error.
when doing a simple mount using rc, it works but when i try to add
vfsOpt='{"CacheMode": 2}', i get the error below.
i simply copied your example from up above.
might be a formatting issue difference between windows and linux or JSON issue.

2020/07/24 10:13:24 DEBUG : rclone: Version "v1.52.2-240-g8f9d5af2-beta" starting with parameters ["C:\data\rclone\versions\rclone-beta-latest-windows-amd64-vfsopt\rclone.exe" "rc" "mount/mount" "fs=wasabi01:1gbtest" "mountPoint=t:" "vfsOpt='{CacheMode:" "2}'" "--log-file=C:\data\rclone\scripts\rr\other\mount\log.mount.wasabit.txt" "--log-level=DEBUG"]
2020/07/24 10:13:24 DEBUG : 2 go routines active
2020/07/24 10:13:24 Failed to rc: no '=' found in parameter "2}'"

Yes windows doesn't understand single quotes. You need to use double quotes and escape the quotes within by doubling them up, something like this I think

vfsOpt="{""CacheMode"": 2}"