Rclone with small vfs cache, question about file removal

Dev Discussions
1

Hello. I have limited disk space and run a vfs cache of about 50 GB in nvme in front of spinning rust. This is a cache for nginx cache files, and nginx is caching large files in chunks using the slice module. One single large file of these could be larger than the entire rclone vfs cache (this is important!). The remote is a local network SMB mount.

While pulling a file from the remote and after the rclone cache is full, I see these messages in the logs:

Jul 26 09:01:36 nginx-proxy rclone[314]: vfs cache RemoveNotInUse (maxAge=0, emptyOnly=false): item e/65/fdd3cdf5263a6eabbcb098565d18a65e was removed, freed 33555325 bytes
Jul 26 09:01:36 nginx-proxy rclone[314]: 5/9c/d90ab1a48fbef466db0685218366a9c5: vfs cache: removed cache file as Removing old cache file not in use
Jul 26 09:01:36 nginx-proxy rclone[314]: vfs cache RemoveNotInUse (maxAge=0, emptyOnly=false): item 5/9c/d90ab1a48fbef466db0685218366a9c5 was removed, freed 33555325 bytes
Jul 26 09:01:36 nginx-proxy rclone[314]: 3/a9/9f74f73760b652ce59276b3b6a10fa93: vfs cache: removed cache file as Removing old cache file not in use
Jul 26 09:01:36 nginx-proxy rclone[314]: vfs cache RemoveNotInUse (maxAge=0, emptyOnly=false): item 3/a9/9f74f73760b652ce59276b3b6a10fa93 was removed, freed 33555325 bytes
Jul 26 09:01:36 nginx-proxy rclone[314]: vfs cache: cleaned: objects 1602 (was 1622) in use 384, to upload 0, uploading 0, total size 49.978Gi (was 50.603Gi)
Jul 26 09:01:37 nginx-proxy rclone[314]: 2/49/36665a58c3dfebb3d661378aa63ae492.0000001075: vfs cache: renamed in cache to "2/49/36665a58c3dfebb3d661378aa63ae492"
Jul 26 09:01:40 nginx-proxy rclone[314]: 5/a7/a41ca960d56930e9c1978094ba145a75.0000001076: vfs cache: renamed in cache to "5/a7/a41ca960d56930e9c1978094ba145a75"
Jul 26 09:01:43 nginx-proxy rclone[314]: 5/91/47f414edcf2ef884240f3ca508353915.0000001077: vfs cache: renamed in cache to "5/91/47f414edcf2ef884240f3ca508353915"

If you look at the vfs cache stats "in use", 384, and this continues to creep up the longer a source file is read. Nginx does not seem to let go of the chunks until the source file has been fully read, or the client has stopped reading it, at which point, rclone sees that the files have been closed and proceeds to upload them with the default 5s timer. But while this is happening, rclone is (correctly) removing files from cache to maintain the maximum 50GB max cache size.

The question is what happens if it gets to a point where the only files that remain in the cache are that of cache chunks of the file that is currently being read. What does rclone do in that case? Will it be unable to remove them (since they are still open by nginx), leading to cache growing beyond max-size? Will it write them back and then delete them? Will they be deleted from cache without ever being written back? Something else?

Thanks!

2

hi,

--vfs-cache-max-size is a soft limit.
if a file is in-use, rclone will not remove it from the vfs file cache.

3

Thanks! So in theory it would just grow indefinitely and trim when possible.

After doing more testing it seems that nginx will close the files after 1000 or so in use files, so this isn't the issue I thought it was: 1000 open files at 32MB per file is 32 GB, so in theory it should never go above the size of the rclone cache. I can always lower the chunk size if I see it go over 1000.

EDIT: also, looks like nginx will respect the user's ulimit, which in my case is set up to 1024 (that would explain the files being closed around 1000). Lowering this limit should also be an option to free the files and upload them earlier.

4

yes, and might want --vfs-cache-min-free-space

5

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.