Large object on rclone mount that exceeds local fs storage capacity

[TrevorKSmith]

Is it possible to have files opened for read-write on a rclone mount that exceeds the local filesystem with cache enabled?

I see this question was briefly answered by Nick here in a similar question.

I bring this up again because it seems that chunking in now available for rclone, although its a beta feature. https://rclone.org/chunker/

If chunking was enabled on a rclone mount, would one be able to open a file with read-write in cache mode, and rclone would only cache the "chunk" thats currently being opened? Hence, i would be allowed to create a 2TB file and open it for read-write with no issues, even though my local fs is much smaller?

[Animosity022]

Cache mode requires a sparse file and that sparse file needs to be able to hold the 'full file' on the file system so you can't exceed local capacity.

The cache mode determination was to go away from chunks and use sparse files and that's a downside as you need the space.

[ncw (Nick Craig-Wood)]

@Animosity022

Cache mode requires a sparse file and that sparse file needs to be able to hold the 'full file' on the file system so you can't exceed local capacity.

This is 100% true for read-write files.

However if you are just reading files and provided the cache directory is on a file system that supports sparse files (most file systems except for FAT32, exFAT) you can read file bigger than free disk space.

Rclone only brings the chunks in it needs and if the file system gets full it will delete the cache file and re-create it flushing all the chunks from it.

So if you open a big file and read little chunks from it, it will work perfectly.

If you open a big file and read it all, then rclone will fill the file system, then delete the file etc.

If you open a big file and write to it, then rclone will need to bring the whole file onto disk in order to save it, so that won't work.

In theory rclone could be cleverer about this with chunker but it isn't yet.

[Animosity022]

@ncw

However if you are just reading files and provided the cache directory is on a file system that supports sparse files (most file systems except for FAT32, exFAT) you can read file bigger than free disk space.

Rclone only brings the chunks in it needs and if the file system gets full it will delete the cache file and re-create it flushing all the chunks from it.

That's good to know as I was just testing it out and does seem to work ok for reading and it has some bumps when it runs out of space, but seems not too bad. I was testing with a 2G tmpfs to see what would happen and I could copy a large file with only a 2GB mount.

Not sure I'd stream with something like that, but that's a great fix.