If I create a union of a local folder in my disk and a google drive folder, and want to to use the new vfs cache, will it double cache the things in my local folder?
It doesn't make sense for me to use the vfs cache for something that is already in my disk, so I'd suggest that the vfs cache by default should ignore remotes with type = local
I ran into a similar conundrum which I solved by creating a vfs mount along side my local directory and then I union the two together as if they were both local mounts.
It looks something like this:
I first mount my crypt to: /mnt/disks/local_vfs with this mount command:
rclone mount --allow-other --dir-cache-time 999h --poll-interval 15s --cache-dir=/path/to/cache/location --vfs-cache-mode full --vfs-cache-max-size 220G --vfs-cache-max-age 168h --rc --syslog crypt: /mnt/disks/local_vfs &
This mount will now utilize the vfs cache when files are read from it. (even though they are being read through the union mount. shown later)
I then union together that along with /mnt/user/local as shown below. (I mounted the local_vfs as read only because I do not ever wish to write to this mount, I use a script with "rclone move" to push the files back to the cloud directly to my crypt:)
[union]
type = union
upstreams = /mnt/disks/local_vfs:ro /mnt/user/local/
action_policy = all
create_policy = all
search_policy = all
cache_time = 120
The mount command I use for the union is simply:
rclone mount --allow-other union: /mnt/disks/local &
I just point my dockers to /mnt/disks/local and they will read files from both /mnt/disks/local_vfs as well as /mnt/user/local and only cache files read from the cloud mount point through the union.
This is adding another layer on top, it works but I'd prefer having less layers not more.
I'd probably use mergerfs if I can't do this with rclone only. But smart solution
If your goal is to do things in as few steps and processes as you can, bringing in mergerfs adds a second layer of complexity to the equation, which you're trying to avoid. Using the rclone union mount is essentially the replacement of mergerfs without installing and configuring mergerfs.
Secondly, you could bypass the vfs caching and union the local dir with the cloud mount directly.
upstreams = crypt: /mnt/user/local/
Obviously the downside is skipping out on the caching. The only way to currently utilize the caching is by mounting the remote to a local dir with vfs commands.
Yes it will.
This is moderately tricky to solve, but maybe you are right, rclone could do something special if the object was local... I guess the problem is that with the union we don't know that the file will be uploaded to the local part of the union without internal knowledge of the union.
In my case, the union is read-only anyway, but I understand it's difficult
A read-only union actually makes it easier I think...
For a reading, rclone can easily tell whether the object is local or not. If it is local then it could "see through" the object to the underlying file and just give the user that. That might not be too hard and would be a great benefit to unions of local + remote storage...
I think we don't need to overcomplicate too much.
The union already knows which remotes are in the union right? So it could be just something like this:
If remote X in union is type = local, then the union will just bypass the vfs cache for requests to that remote
vfs caching would almost need to be setup as a per remote basis for this to operate in such way. Instead of it being configured at the time the remote is mounted.
This would eliminate my need to mount a remote for the sole purpose of caching the reads from the read only mount.
That is more of a VFS union, so moving the union code to the VFS layer. That might be possible too.
I think for reading objects it should be possible to look through the object and see the local file.
This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.