Rclone VFS feature as a persistent cache for large game pool?

Help and Support
1

Hi,

I'm currently searching for a solution to allow for a specific configuration and I believe that rclone might be a viable option, so I wanted to see if those that have experience using the relevant features of the rclone mount could let me know if they think it's a good fit for this use-case or let me know if there's any caveats I may not have thought of or should be aware of. Additionally, I have some questions about specific configuration options and how they come into play for this theoretical setup.

I have browsed the docs and forms a bit, but I ask these questions due to being uncertain if my interpretation of some of this is correct.

Goal:
Allow a remote client to access a huge pool of game files (~40TB) remotely in a transparent fashion with more limited local storage. Since the files are for games and will require high performance, low latency access.

  • TrueNAS SCALE server, Windows 10/11 client
  • Transparent access to files from the server/cache with automatic caching on first read and when files are changed
  • ~4TB drive on the client used entirely for caching files from the remote server
  • Cached files are kept "forever" and only cleared to make room when the cache reaches its size limit
  • Ideally files are purged from the cache in order of the oldest "last accessed date"

Obviously I can handle configuring the server and dealing with that side of things is outside the scope of this post. I likely will start with an externally available SFTP server that's accessed via pre-shared key, or some other rclone supported protocol through wireguard.

When it comes to the client that will run rclone, from what I've read it seems that rclone's mount feature can fulfil the requirements listed above and serve my overall purpose with --vfs-cache-mode set to full. In general, I'd appreciate feedback on the feasibility of this from the rclone side of things as aforementioned.

More specifically:

  • Can --vfs-cache-max-age be set to zero/a negative value, or otherwise disabled so that items are only removed from the cache due to --vfs-cache-max-size being reached. If not, what is the maximum value that can be specified for it (i.e. can I specify months, or even years?)
  • I see that rclone will only download files from the server in chunks when the file isn't in the cache instead of downloading the entire file. While this makes sense in many cases, in mine I'd actually prefer the opposite due to the nature of the files and need for minimal latency during random access. I know --vfs-read-ahead can influence this and I could, for example, set the value of that to be larger than the largest game file I plan to have accessed to ensure maximal read-ahead; however, given that it's called "read-ahead" I imagine that no matter how large of a value I set rclone will only download extra data that's beyond the initial read offset and that bytes coming before the offset will not be preemptively downloaded. If this is true, is there anyway I can assure that the entire file is downloaded before the VFS signals to the request that it has been "fulfilled", so that the application will effectively stall until the entire file is ready? Or if not, at least a configuration where the entire file will always be downloaded and cached upon a first read of any kind even if the application receives the data at the same time?
  • Are there any kind of scaling issues that will prevent rclone from operating effectively with a cache this large (again, ~4TB)?

If it matters, the contents of the server will change only very rarely, so 95% of the operations will be reads.

The overall idea is having a huge library at the client's disposal remotely and with great convenience. You go to play a game for the first time, run the application that needs the file(s) and they're downloaded from the network, with the game proceeding once they're finished (or slightly less ideally as they're being cached). Then in the future when the game is started it is always served from the local cache unless it hasn't been played in so long that the file(s) were removed due to cache capacity limits. Essentially, something that automates the step of first downloading the game on demand before it's played and removing it if it hasn't been used for a long time and space is needed.

I am slightly concerned the overhead due to data management (i.e. moving between server, RAM, and disk cache, etc) being an issue due to the somewhat latency sensitive nature of the files even once they're cached. Not sure if someone can chime in on how much performance from the cache differs vs direct from disk as obviously if it's small enough then it won't matter. I originally considered just using Window's Offline Files, but have only heard terrible things about it and was specifically advised against using it for this application, which is a shame because in theory it sounded perfect.

Thank you for your time.

1 Like
2

hello and welcome to the forum,

rclone does not do that. if you enable cache for reads, rclone will cache the first read, second read, every read.

in the past, did testing and --vfs-cache-max-age=0 did not work as you want.
that each time --vfs-cache-poll-interval expires, rclone will pruge stale objects from cache.

as for values, that is documented at https://rclone.org/docs/#time-option
so try --vfs-cache-max-age=9999y

the cache uses chunk reading.
if you have a 100GiB file and read just the first byte, rclone will download only a single chunk, not the full 100GiB file

3

I have the same doubts as you. Also have large pipe now and large local storage, and I know that everything downloaded will be read again, so I'm still trying to find the mix to where files are downloaded to disk as fast as possible but without killing the drive.

Ideally I would like to download as much as possible, because i have a large download speed, no limits, and low latency. And buffer as much as possible to disk because I also have enough storage and want next reads to go from disk.

At same time ideally RAM would be used to ensure that disk i/o is not overwhelmed with just writes so that streams from disk are not affected by the writes.

1 Like
4

rclone does not do that. if you enable cache for reads, rclone will cache the first read, second read, every read.

Sorry, that was supposed to say 'with', which as you noted is how rclone behaves. I have corrected the OP.

in the past, did testing and --vfs-cache-max-age=0 did not work as you want.

Thanks for letting me know. Although it's not as clean, this makes it clear I can at least specify a massive maximum that effectively means files will never be purged due to age

if you have a 100GiB file and read just the first byte, rclone will download only a single chunk, not the full 100GiB file

I do understand this, so I guess there is no alternate mode where I can force it to download the whole file upfront; however, this still leaves my question about the read-ahead option, which may allow me to get close to the desired behavior. In almost all cases the file will first be read from the very beginning (i.e. offset 0x0) and I imagine if the read-ahead size is set to be larger than the file size that the entire file will be downloaded (in chunks) right away.

5

Pretty much in the same boat, and a little surprised that something that satisfies things roughly within this purpose aren't a bit more common, though I recognize it's somewhat niche.

I'm basically just trying to avoid having to keep two 20TB hard drive in a particular machine and figure the sooner that I can get something like this working the better, as it should essentially be sufficient forever and passively scale as network speeds are increased and I upgrade the hardware of my server for other reasons anyway, other than maybe upgrading the cache drive on the client.

OneDrive (and as I mentioned Offline Files) are basically what I desire, but OneDrive isn't self hosted and Offline Files is basically broken as stated early. At the end of the day I'm sure this can be done really well with some kind of purpose-built, commercial solution, but of course that's not really feasible/desired.

6

In my case the files I'm downloading are 1~3gb which should take 1-3s to download from a 10 gbps backend with no limits, so it would really make sense to download the entire file in one go and then send from disk.

But at same time we need to be careful to not kill the disk with too much i/o so I don't really know what is the best mix yet.

I have been trying this:

  --vfs-cache-max-age=8766h \
  --vfs-cache-max-size=14.1T \
  --vfs-cache-mode=full \
  --cache-dir=/mnt/cache \
  --vfs-read-chunk-size=64M \
  --vfs-read-chunk-size-limit=512M \
  --buffer-size=20M \
7

whatever file, on the mount, you want to force into the vfs file cache, run md5sum on it.
that will force rclone to cache the entire file.

too many variables involved that are specific to your system.
at this point, why not run a rclone mount command, do a test, see what happens?

8

I was thinking and what we want to emulate is how to get back the cache from where a time rclone would download the whole file before letting the user read, which would slow down opening the file.

Setting --vfs-read-chunk-size to 0 or "off" disables chunked reading.

This would be the behavior expected with this flag right? I thought about using that but then even if it happens only once, if a delay of 2-3 seconds can be avoided for the end user i think that should be pursued.

--vfs-read-chunk-size 0 off + --buffer-size=64M what would happen when a user open a files?

Will the 64 mb buffer in ram allow for fast opening of the file, while downloading the entire file to disk in background ?

Even myself still feel confused how all those options work together

9

Forcing a checksum on a file makes sense, but that completely breaks the transparent access of the setup if I have to go out of my way to do that with a file before using it for real.

Yes, at this point I will just give the software a spin as there don't seem to be any obvious significant issues, though of course there are still plenty of details that need to be hashed out. I have to setup how the files will be served from the host first.

@random404 I missed that you can disable chunked reading? Slightly confused by your comment about it, are you saying you haven't tried it personally yet? I too would assume that turning it off would cause the whole files to be downloaded (and therefore cached) when trying to read any portion of the file, though confirmation on this would be nice. Eventually I will test it myself.

10

I just tested three different combinations

  --vfs-read-chunk-size=1M \
  --vfs-read-chunk-size-limit=8M \
  --buffer-size=1M \

  --vfs-read-chunk-size=0 \
  --vfs-read-ahead=2g \
  --buffer-size=20M \

And

  --vfs-read-chunk-size=64M \
  --vfs-read-chunk-size-limit=512M \
  --buffer-size=20M \

You can create the 3 mounts at same time and test with time cp file /dev/null. I have found that 0 read chunk size is the fastest to get the file on-disk at the cost of higher bandwidth usage.

The other options have more balance in network usage vs disk usage. Try doing the same and report back what you think is the best ! I'd probably run with chunk size 0 if I had confirmation from @ncw that users can still start streaming the file before it's fully downloaded

11

Now that I was thinking about it, how does this settings relates to the users downloading from the mount ?

Does any of these settings impact in how fast can the user download from mount > user.

Not talking about source > mount. But rather if there is any impact into mount > user beyond the speed of the source to deliver the data

12

might try to confirm without involving ncw.
perhaps set --bwlimit to a low value, and start to stream, see what happens.

13

This disables the chunked reading of files. What that means is that files are opened at the cloud provider as-if they are going to be read to the end.

This means that the cloud storage provider buffers up the whole file to send, and may charge you for the whole file even if you don't download it all. Depending on the storage provider you need to watch out for this! I believe Google drive will count the whole file against your download quota even if you don't download it all (which likely rclone won't).

Depending on what the user does rclone may not read the file to the end. For example if the user seeks in the file. In this case the setting has cost more bandwidth and potentially more money/quota at the cloud provider.

Yes it makes no difference to this.

This chunk size isn't buffered in memory, all it is doing is changing the size of the request to the cloud provider. Using a smaller chunk size will definitely use more transactions each of which takes time.

I would have thought using something like --vfs-read-chunk-size 1G --vfs-read-chunk-size-limit 1G would work as well as setting it to 0.

14

We are talking about our glusterfs backend here, there are no limits or quotas.

The only limits are the limits of our hardware !!

15

Whee! Go large or infinite then then :slight_smile: This is the best case scenario for --vfs-read-chunk-size 0.

I just thought I'd explain the rationale behind those flags!

16

With chunk-size 0 how buffer-size works together for the end user? What would be the optimal value ?

17

Buffer size is for read ahead. If your source is glusterfs then try --buffer-size 0 or small 16M

18

Would rclone use buffer size for files already in cache?

Do you still need rclone read ahead if you have the read in chunks flag?

--buffer-size=20M \
  --vfs-read-chunk-size=64M \
  --vfs-read-chunk-size-limit=512M \

I have been testing this and it seems fine. But I am still not sure if it's optimal. I'd like to have all downloaded data flushed to disk even the client disconnects before fully caching the file.

And I wouldn't like to buffer into ram reads after the file is on disk because i use zfs arc

19

No it wouldn't.

These perform different functions. The --buffer-size buffers in local RAM whereas you can think of the --vfs-read-chunk-size signalling the source that you are going to read that much data.

Rclone will flush the --buffer-size of data, even if the client hasn't read it, to disk.

Sure. After the file is on disk, rclone just reads chunks from it and hands it straight to the OS - very little buffering involved.

20

Does any of the settings affects the speed at the end user ( remote > server > end user ) can download from the mount ?