When rclone mount is run using S3 as the backend, I want to start automatically loading data to the local VFS cache so files open faster without needing to be downloaded.
Is this possible already? A caveat is that if I try to open a file on the mount that has not already been pre-loaded in the cache, rclone should start loading this file instead of pre-loading other files.
Does that make sense? Rclone doesn't know which files it should preload and then it loads something and it's wrong.
Not sure how you are mounting your S3 backend, Assuming Linux, you can create a script to run after it's mounted and to cat the file and send it over to /dev/null so it will be cached if you have enough cache size. If your cache size and life is long enough, file should remain there for a long time as well as survive a remount.
that PR applies to the vfs dir cache, not the vfs file cache.
i have done that, works well.
might try setting --vfs-read-ahead to a large value.
Thanks for your replies. I thought of using --vfs-read-ahead and setting it to the approximate size of my backend store (100GB). My worry is that if I do that and load a file, it will load the next sequential 100GB, even if a file at the other end of the store is opened, meaning I'll have to wait for all 100GB to be loaded before I can open the file. Is that accurate?
I'm using Windows, but any chance you could share your script if you have one?
@asdffdsa I checked out --vfs-read-ahead, it seems to only read ahead for the files that are being loaded, not for other files sequentially saved ahead of them. Is that correct?
this has been discussed in the forum a few times, here is an example i posted.
https://forum.rclone.org/t/pre-load-specific-folder-in-mounted-drive/29861/4?u=asdffdsa
that could be true.
This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.