Hi there, I use the VFS cache feature (writes mode) with
rclone mount to get access to cloud storage such as Google Drive and Backblaze B2 with rclone's encryption as a file system mount on my Linux computers. I do this to run borg with its repositories on the rclone mount for my daily manual offsite backups, and it works wonderfully. The only problem is that my upload speed (20Mbps) is vastly slower than my IO speed (2000Mbps), so the VFS cache fills up very quickly. The problem is at it's biggest when I do the initial archive creation, where I backup all my data from the start without any deduplication (I will effectively need to store a copy of the total backup size to cloud storage on disk tepmorarily). There is
--vfs-cache-max-age which I can set to 0 seconds and
--vfs-cache-max-size which I can set to 0 bytes so that it only stores the file on disk when its still uploading, but this doesn't solve the problem because the files are still open because they're still uploading, and the disk speed is way too fast compared to the upload speed so the VFS cache fills up with open files that can't be deleted yet. Even then, I still think this should be addressed because I might want to backup a large amount one day (maybe I took a lot of photos or downloaded a lot of Linux ISOs that day).
I would like to suggest a feature to solve this problem. I think if the write speed to the VFS cache can be rate-limited by rclone to match the upload speed (similar to
--bwlimit), this would solve the problem a bit. This could probably be done without rclone in Linux by limiting disk speed with schedulers or cgroups, but it's still difficult to do because it can most likely only be done on disks themselves and not directories, so if you have only one disk connected to the system you would slow down the whole system, maybe unless you do something like make a loop block device. It would be more practical IMO to have this feature in rclone if possible.
Let's say there is now a feature to do this (eg.
--vfs-bwlimit write_speed:read_speed) so we set
write_speed to my upload speed (20Mbps). It will be okay, until the upload speed decreases (eg. something else on the network is using bandwidth, so my actual upload speed decreases to 10Mbps), or maybe even the internet cuts out for a minute making the mount freeze (but maybe the current write will still pass through VFS and therefore VFS cache, but I'm not sure). In the former case, the problem will occur but in a much smaller scale (not ideal but vastly better than currently). In the latter case (assuming current writes pass through VFS when there is no internet), it would be even worse (disk size increases by 20Mbps, but still not as bad as 2000Mbps without limiting).
So, I think this could be solved if rclone can calculate the current upload speed periodically (eg. set poll interval with
--poll-upload-speed, maybe 1 or 5 seconds is a good default) and then dynamically set the hypothetical
--vfs-bwlimit option while the program is running. Rclone can already limit bandwidth speed, so it seems like it's possible to implement. In the meantime, maybe there's some way by doing the loop block device thingy and write a script to detect rclone's upload speed and change the IO speed of the disk or block device, but what I said earlier about it being more practical IMO to implement this in rclone itself applies here.
Maybe there's another way to solve this problem that I haven't thought of, if so, please state in the comments. I'm looking forward to your thoughts and possible implementation in rclone if it seems like a good idea.