Absolutely! It hovers around the 130,000K mapped memory mark and stays there most of the time with normal work file usage. As long as no enumeration of multiple levels of directories takes place then its great.
Difficult to say anything from 10-1000 on average I'd say. File sizes will vary between 200MB-18GB. A system that has been running in excess of 1 year with this setup is currently up to 3.4GB mapped memory for rclone. It seems to have topped out there and now just permanently uses that RAM.
This would explain something I definitely noticed. A freshly booted system will use about 137000K and will then about double during standard use. The problem is it never frees the memory again after getting it even after having been idle for hours. I would have assumed it to be returned to the system after a fair amount of timeout. This situation is true for both a scenario where several directories were accessed to work with files as well as where only a single directory was accessed and multiple large files added to that directory once a day.
So it's not just about the expired directories holding on but also expired file locks it seems.
How expensive is this transaction to the system (CPU, I/O access)? If the hit is low then do it more frequently as it will keep the tree nice and fresh and obviously clear its memory footprint regularly. If the hit is moderately high then rather wait for a higher count as per your suggestion. EIther way it will definitely make an improvement to the current situation
Go isn't brilliant at returning memory to the system. Under Linux it marks it as no longer in use, then the kernel can reclaim it if it is under memory pressure.
You can see if this is the case by running the memory profiling - go tells you how much memory it is using and you can compare that to the system usage.
Filling up the directory cache is quite expensive in terms of round trips to the cloud provider
However there is a flag for expiry time so I thing rclone really should be ditching the memory and users can increase that flag if they want.
This would be really helpful. My mount just got killed by the kernel due to the system being out of memory. It was using 8 GB of memory with ~ 2 million objects, most of which were not being accessed frequently.
Do you think setting GOGC to a lower value would help with this in the meanwhile?
