Use case: a persistent synchronized mount cache

[kforner (Karl Forner)]

I'd like to know from the rclone community wisdom if rclone could support my use case.

Suppose you have a bunch of data files in some remote storage folder, some useful for some analysis some not, but you really have no way to predict which files will actually be accessed.

What I would like is:

• a virtual file system (mount) that exposes these files locally
• files are (seamlessly) downloaded lazily, when accessed
• files are cached locally in case the code read a file twice, or if the analysis is re-run
• the cache is persistent, so that files used today won't have to re-downloaded tomorrow (if I restart the system , e.g. VM, docker container, mount)
• there is some kind of cache validation, in case files have changed on the source storage. This validation does not have to happen at each access, but when last validation is too old. This validation could simply use size/modification time metadata.

From my previous experience with rclone VFS I believe that rclone can support this use case, but I'd like a confirmation, and possibly some hints and advices.
Also the reason I'm asking is that I first considered blobfuse2 for this task, since the current storage is an Azure blob container, and the developer told me that the persistent aspect of this use case it not supported by blobfuse2.

Regards,
Karl

[kapitainsky]

@kforner

a virtual file system (mount) that exposes these files locally

rclone mount

@kforner

files are (seamlessly) downloaded lazily, when accessed

Yes. Check VFS chunk reading for details

@kforner

files are cached locally in case the code read a file twice, or if the analysis is re-run

This is what VFS files caching provides

@kforner

the cache is persistent, so that files used today won't have to re-downloaded tomorrow (if I restart the system , e.g. VM, docker container, mount)

--vfs-cache-max-age controls how long it is valid. Can be 1min and can be 1 year.

@kforner

there is some kind of cache validation, in case files have changed on the source storage. This validation does not have to happen at each access, but when last validation is too old.

check --dir-cache-time or you can always trigger it manually if some other logic is required

I suggest you simply start something and then come back if it does not work as you expected.

[kforner (Karl Forner)]

Thanks @kapitainsky
That confirms what I thought and read. I guess I wanted to check if this use case made sense. From your answer I get that's not absurd.

[kapitainsky]

As always no solution tick all boxes precisely so for sure you will discover some rough edges. Looks like you know exactly what you want so should not be a massive undertaking to give it a try, test and then tweak what possible.