I'm currently searching for a solution to allow for a specific configuration and I believe that rclone might be a viable option, so I wanted to see if those that have experience using the relevant features of the rclone
mount could let me know if they think it's a good fit for this use-case or let me know if there's any caveats I may not have thought of or should be aware of. Additionally, I have some questions about specific configuration options and how they come into play for this theoretical setup.
I have browsed the docs and forms a bit, but I ask these questions due to being uncertain if my interpretation of some of this is correct.
Allow a remote client to access a huge pool of game files (~40TB) remotely in a transparent fashion with more limited local storage. Since the files are for games and will require high performance, low latency access.
- TrueNAS SCALE server, Windows 10/11 client
- Transparent access to files from the server/cache with automatic caching on first read and when files are changed
- ~4TB drive on the client used entirely for caching files from the remote server
- Cached files are kept "forever" and only cleared to make room when the cache reaches its size limit
- Ideally files are purged from the cache in order of the oldest "last accessed date"
Obviously I can handle configuring the server and dealing with that side of things is outside the scope of this post. I likely will start with an externally available SFTP server that's accessed via pre-shared key, or some other rclone supported protocol through wireguard.
When it comes to the client that will run rclone, from what I've read it seems that rclone's
mount feature can fulfil the requirements listed above and serve my overall purpose with
--vfs-cache-mode set to
full. In general, I'd appreciate feedback on the feasibility of this from the rclone side of things as aforementioned.
--vfs-cache-max-agebe set to zero/a negative value, or otherwise disabled so that items are only removed from the cache due to
--vfs-cache-max-sizebeing reached. If not, what is the maximum value that can be specified for it (i.e. can I specify months, or even years?)
- I see that rclone will only download files from the server in chunks when the file isn't in the cache instead of downloading the entire file. While this makes sense in many cases, in mine I'd actually prefer the opposite due to the nature of the files and need for minimal latency during random access. I know
--vfs-read-aheadcan influence this and I could, for example, set the value of that to be larger than the largest game file I plan to have accessed to ensure maximal read-ahead; however, given that it's called "read-ahead" I imagine that no matter how large of a value I set rclone will only download extra data that's beyond the initial read offset and that bytes coming before the offset will not be preemptively downloaded. If this is true, is there anyway I can assure that the entire file is downloaded before the VFS signals to the request that it has been "fulfilled", so that the application will effectively stall until the entire file is ready? Or if not, at least a configuration where the entire file will always be downloaded and cached upon a first read of any kind even if the application receives the data at the same time?
- Are there any kind of scaling issues that will prevent rclone from operating effectively with a cache this large (again, ~4TB)?
If it matters, the contents of the server will change only very rarely, so 95% of the operations will be reads.
The overall idea is having a huge library at the client's disposal remotely and with great convenience. You go to play a game for the first time, run the application that needs the file(s) and they're downloaded from the network, with the game proceeding once they're finished (or slightly less ideally as they're being cached). Then in the future when the game is started it is always served from the local cache unless it hasn't been played in so long that the file(s) were removed due to cache capacity limits. Essentially, something that automates the step of first downloading the game on demand before it's played and removing it if it hasn't been used for a long time and space is needed.
I am slightly concerned the overhead due to data management (i.e. moving between server, RAM, and disk cache, etc) being an issue due to the somewhat latency sensitive nature of the files even once they're cached. Not sure if someone can chime in on how much performance from the cache differs vs direct from disk as obviously if it's small enough then it won't matter. I originally considered just using Window's Offline Files, but have only heard terrible things about it and was specifically advised against using it for this application, which is a shame because in theory it sounded perfect.
Thank you for your time.