Rclone union cache_time

Help and Support
1

First of all thanks for launching this - been looking forward to this for a long time.

I'm trying to understand the cache_time option. I currently merge a local folder with several remotes using mergerfs with this command:

mergerfs LocalFilesLocation:RcloneMountLocation MergerFSMountLocation -o rw,async_read=false,use_ino,allow_other,func.getattr=newest,category.action=all,category.create=ff,cache.files=partial,dropcacheonclose=true

Does cache_time work the same as cache.files or should I leave it at the default 120? What exactly is cached - just the amount of free space i.e. not relevant for a unlimited gdrive and a local folder which the local system manages?

Thanks in advance.

2

I just created the following union remote:

[tdrive_union]
type = union
upstreams = /mnt/user/local/tdrive_vfs remote0: remote1: remote2: remote3:
action_policy = all
create_policy = ff
search_policy = ff

and mounted it:

rclone mount --allow-other --buffer-size 256M --dir-cache-time 720h --drive-chunk-size 512M --log-level INFO --vfs-read-chunk-size 128M --vfs-read-chunk-size-limit off --vfs-cache-mode writes tdrive_union: /mnt/user/mount_mergerfs/union_test

All my existing files appeared correctly, but when I created a new folder in /mnt/user/local/tdrive_vfs it didn't appear in the union mount.

Help please as the launch times for media playback are good and seem better than mergerfs.

3

ok figured out the --dir-cache-time 720h applies to the local folder in the union as well. Is there a way to get changes made locally picked up quickly, but gdrive changes still will a long cache time? I don't write direct to gdrive (upload via rclone).

4

That is a sensible suggestion but it isn't possible with rclone just yet... The caching happens at the VFS layer and the union is made at a lower level. What you really want is a union of VFSes which I've had some discussion about on the forum already so I've got in in my thinking list!

5

Hi @ncw sorry for the slow reply, but I've been swamped at work.

Is there a way with union to get changes made to local folders to show up 'immediately'? I think for my scenario changes made to the rclone mounts/remotes will get picked up as I only upload to gdrive via rclone. But, it's the lag I've got at the moment for local folder changes that's stopping me using this.

A bit more on my workflow which I think is a common one.

  1. List item I have a union of a local folder and various rclone remotes

  2. List item New items get added to the local folder

  3. List item A rclone upload script periodically moves files from the local folder to the various remotes

Edit

does union support poll-interval? I just got this error:

2020/06/13 10:32:34 INFO : union root '': poll-interval is not supported by this remote

Yet https://github.com/rclone/rclone/issues/2837 says it does?

6

What is required is for the local backend to support change notify.

There is an issue about this here

Ideally rclone would use fsnotify or similar but they are all annoyingly OS dependent.

Probably your best bet right now is to use the rc and vfs/refresh to refresh the folder that changes happened in. You might even be able to script this...

I think all the underlying remotes need to support change notify for it to support change notify.

This may be a design error - maybe it should only need one remote to support it before enabling it.

7

Thanks. I think you are right that currently all the remotes need to support notify - when I made changes to a mounted remote i.e. via rclone, the changes still weren't picked up by the union.

I hope this gets fixed as union is unusable for my scenario where files are being changed constantly locally and to rclone mounts within the union, which is a shame as media playback seems faster with union than with mergerfs based on my limited testing

8

How many files do you have locally? If you don't have too many then maybe we could implement a polled system to detect changes.

Hmm, I could probably write a bash script which polled with rclone and sent VFS refresh commands too.

I'd like to implement proper changenotify eventually but that could be a workaround for now.

9

Probably too many files, plus the count isn't static as they are constantly being uploaded/added to:

root@Highlander:/mnt/user/local/tdrive_vfs# find . -type d -or -type f | wc -l
60766
10

Yes that is quite a lot. How many directories do you have? If we implement an inotify scheme we'll need a handle per directory...

11 
root@Highlander:/mnt/user/local/tdrive_vfs# find . -type d | wc -l
5396

Thanks for looking to help - again!

1 Like
12

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.