Mount with VFS settings questions

I have some questions about some of the settings for rclone mount using VFS. I was using cache initially but that was a memory hogger so switched to just VFS. I read what each setting does in the rclone docs and checked the forum but I still feel a bit uneasy.

rclone version: 1.54
OS: Debian buster (10)
arch: amd64
remote: Google drive (with SA)

mount service:

Type=notify
ExecStart=/usr/bin/rclone mount \
        --allow-other \
        --fast-list \
        --dir-cache-time 12h \
        --poll-interval 30s \
        --buffer-size 8M \
        --cache-dir /opt/rclone/vfs-backend \
        --vfs-cache-mode full \
        --vfs-cache-max-age 12h \
        --vfs-cache-max-size 242G \
        --vfs-cache-poll-interval 1m \
        --no-modtime \
        --vfs-read-chunk-size 32M \
        --vfs-read-chunk-size-limit 0 \
        --transfers 4 \
        --log-file /opt/rclone/rclone.log \
        google_with_crypt: /srv/path
ExecStop=/bin/fusermount -u /srv/path
Restart=on-failure
RestartSec=20

The following are the ones that I am not sure about, along with what I believe they do:

--fast-list this really makes a big difference on the amount of calls but it has no effect on mounts?
--dir-cache-time controls how often rclone will ask the remote for a new listing of the whole dir. Does this work per dir? poll-interval will take care of any new files / deletions in between.
--poll-interval how often rclone asks gdrive for changes. I assume this is for the whole remote and gdrive replies with which files have been added,removed,modified as that would be the benefit compared to dir-cache-time that must use transaction(s) per dir.
--vfs-cache-poll-interval how often rclone checks if the vfs cache contents have exceeded the age or size limits. Lowering the value would make sense only if you have quite small cache-size.
--no-modtime for Gdrive do modtimes take another transaction? If not there is no real reason to use it as there is no performance gain.

any other comments about my mount options are welcome!
Thanks

Does nothing on a mount.

No as dir-cache-time is how long when you do a directory listing on a location, it will last in memory. It keeps the file structure of what are you listing in memory. It's not directly related to polling. Polling checks for new changes and invalidates the directory cache if it detects a change. Only a listing would refresh the directory cache.

IMO, I'd say not really. You can leave it at 60 seconds and regardless it's going to clean up what it can. You really need enough space to keep what you want to stream / use concurrently for best use.

Why would you turn it off? There's really not hit for getting the mod time and it's useful when validating things and needed for certain application to avoid things being marked as new.

Best to just remove / use defaults.

That is the default so you can remove it.

This only keeps items in your cache for 12 hours as depending on your use case, you can make this value very large and let it expire when it fills/needs the space.

Google Drive is a polling remote so you really want this value to be very large. Any changes are handled via polling.

Best to remove and use the default.

1 Like

That's what I had understood. Thanks for confirming.

So the 12h pass, the listing expires and only if you navigate / request something for that dir, it will cause rclone to request a new listing from gdrive.
If that dir is being used by plex/emby/etc the expiry of the listing doesn't affect them, as the moment that they decide to poll X file that will cause a dir relisting.
I am still not quite understanding it. If for example you used inotify on that dir, when the dir-cache-time expires, would inotify catch a delete signal for the contents?

Agreed. What I was thinking was more of an edge case. For example you have 5G available space and you have set vfs-cache-max-size 4G. Then you would want the check to happen pretty frequently to avoid filling the disk (ofc if you have an active file that is pretty big, how frequent the check is would not save you).

In the docs it mentions some remotes that modtime requires an extra transaction per file. I couldn't find if google was part of those remotes. Now I know :slight_smile:

I thought these affected memory usage but found a post from Nick that says they don't, so defaults it is.

Ok, so I 'll set it to a couple of days and --dir-cache-time to longer maybe even some weeks (set in days)? Or that's too large?


Since I am using vfs-full should I also specify --vfs-read-ahead ? Any idea which location it uses to temporary store --vfs-read-ahead ? edit The negative part is that it will use the disk which A. is slower than memory and B. increases the I/O load.

Yes, if the directory cache is expired or not there, it would get a new directory/file listing of the directory to get the metadata about the files.

inotify doesn't work rclone/fuse.

API hits for Google really don't matter too much as you have plenty of API quota per day.

I use 1000H for both.

I see no need for it in my use case. The --cache-dir defines where the cache files are located. I just use a 1TB SSD disk which at current wear will last me 5-6 years which is fine :slight_smile:

I run a rclone delete --min-age X (albeit without fast-list) and in less than 5-10min I kept hitting the 403 error non-stop. With fast-list it sorta managed to delay itself properly, so maybe this is a bad example. I.e not using fast-list might be proportionally worse than not using -no-modtime.

But yea I will remove that flag.

Ok, I 'll go with that as well!

I am using some VPS that sometimes are limited by mem or space or both, that's why I asked but I think I got the gist.


Sounds good, thank you a lot Animosity! Next step will be either union-fs or vfs & writing files -> uploading to remote.

That's a different use case as your OP was about a mount. For any other operations, fast-list should be used. I was only answering what you had asked :slight_smile:

I use mergerfs as I find that to be a bit nicer as I can hard link.

yap, the rclone delete was to comment only to -no-modtime. I just compared it to the performance loss if you don't use --fast-list, sorry that example is anything but clear.

Will check that out

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.