Best way to limit rclone local cache per folder?

I'd like to use the VFS cache but limit how much space a single folder can use...

is the best way to do this still usin a rclone union and then set the local cache for each union as I want to be?

I usually prefer to have less layers for performance reasons, as I expect rclone to use ~40 Gbps out of this mount.

It's a raid 0 of nvme ssd drives

Is that a specific folder, or any folder you'd like to limit?

The VFS cache sits on top of the union so I don't think this idea will work.

Maybe a program which scans the files in the cache via the API and removes the ones you don't want would work?

I'd like to specify a single folder that can't go over X TBs, the other folders can be unlimited

The union wouldn't have a vfs cache only the remotes that it's uniting, wouldn't that work?

Also I have been thinking and there are a lot of good reasons to be able to set specific amount of disk space per folder, or in a single folder. And maybe even specific cache times for different folders or a specific folder with another global settings

Big folders with lots of small files - It would kill performance in HDDs and even in NVMEs and those files would be easy to download on the fly in high speed networks

Rclone cache takes a long time to start if they have a lot of items.

Some folder may have files that are accessed only a few times, while others are accessed a lot more. And those files will take disk space without need.

And probably a lot of other use cases I can't think right now

Lots of things rclone could do with the cache - that is for certain!

Some different cache policies would help - like in this issue: VFS Cache: control caching of files by size · Issue #4110 · rclone/rclone · GitHub

I guess if rclone had a slightly richer API then you could implement some cache cleaning policies externally.

Or maybe have a cache policy plugin.

I like that idea, it seems similar to what I said indeed.

I do think rclone cache needs to expand a lot because it's really useful but to be used in a massive scale like I do it needs to be like a cloudflare cache, you can tweak a lot of stuff.

How we can make that happen ?

Sorry for the delay replying.

I guess if you'd like some specialized features for the cache I could do those in a contract? Would that be interesting?

Can you message me with a form of contact so we can discuss this further?

I'm moving all my servers to 10 gbps and your help would sure help a lot !

Yes sure - drop me an email to discuss further!

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.