Hello. I have a question about rclone with vfs. I want to cache file names / folders / metadata without caching the files' contents themselves is there an easy way to do this? I spent the last few days trying to figure out how but I have not had much luck.
I have tried using full, minimal, and writes but they don't seem to do what I would like them to do.
I am trying to access a few thousands files across a few directories and the files are too large to cache them being upwards on 200gb each. I am doing 3-4 million api calls on dropbox a day across multiple accounts and need low latency for my application to stay responsive. I have a 2-4ms connection to dropbox's api. When I cache using vfs it is okay at first buts eventually starts struggling under high load with something as simple as listing files in a directory taking 10-40 seconds. This makes it so the files in them can not be read until it responds. When there isn't much load listing files is instant and reading those files is pretty quick.
When I turn caching off with the same high work load file access times are lower than with caching on but directory listing times are terrible which are to be expected.
Having some in between setting would be nice that way hopefully vfs stays responsive.
My vfs cache is stored on a zfs pool with 8 drives in a raid 0 configuration and 2 ssds as a cache for zfs. zfs has always performed well. I tried a few different setups and this seems to work fine.
I do monitor the rclone logs and I am not being api rate limited. I have --tpslimit 8 set for each dropbox account.
Oh this clears a lot of things up for me, I didn't know that.
Is this just for directories or does it cache the files as well? Or is that a separate setting?
Ahem, since nobody clicked my link, that solution is exactly what I suggested
@Animosity022 - how come the priming takes so long for you? Mine takes 5 minutes at most (it varies, even though the content doesn't change much). That's with about 55k folders and 286k files.
I just didn't understand what that exactly did from the way you explained it is all. When Animosity022 broke it down it made a lot more sense to me. Thank you though ^-^
This may be a dumb question but I noticed latency goes up when loading fresh data from the remote even if part of the file is in vfs. I noticed as long as the files are being constantly read/refreshed latency stays low. Is there a setting I can use to optimize this? Like if a file is being read to keep it open for longer in memory after it's been closed?
