As per the subject. I'm being forced to run rclone (v1.63.1 amd64 running on Debian11, in case it matters) and its memory usage has been a problem. --use-mmap looks like a pretty painless way to reduce that (just add the option to the command-line and off it goes), but I would like to know what are its possible negative impacts (after all, if it was all upside, it would be the default, right?)
Not including logs, etc as my question don't actually depend on any specifics.
TIA!
I think it is quite simple as per doc:
It is possible this does not work well on all platforms so it is disabled by default; in the future it may be enabled by default.
It was implemented as potentially another way to help memory recycling which in Go is not the fastest. It can work or not depending on platform. Nobody spent too much time on it so it is unsure how safe it is to make it default (and it is clearly documented in docs). And probably will stay like this. The reason being that it is easier to rely on what Go language and runtime provides than to experiments with platform depended tuning. You could ask why Go does not make it default in their runtime?
For the record, I'm using the nmap feature on all my mounts and transferts, on windows, and I have to say that yes, the memory is free'd up faster than without the argument. It's not nigh and day, but If memory is a sparse ressource, try it. No stability problem either with (or without) it.
The same here. I use it often too and have not seen any issues yet. However I would not use it on the system I can not afford to crash.
Thanks for quoting that, I had RTFM but failed to see it.
For the sake of those that will eventually come a-googling, here's the direct link for the above-mentioned part of the docs: rclone docs: --use-mmap option.
It was implemented as potentially another way to help memory recycling which in Go is not the fastest. It can work or not depending on platform.
Reading the docs you referred, it also says:
If this flag is set then rclone will use anonymous memory allocated by mmap on Unix based platforms and VirtualAlloc on Windows for its transfer buffers (size controlled by --buffer-size). Memory allocated like this does not go on the Go heap and can be returned to the OS immediately when it is finished with.
It looks more like a total bypass of Golang's memory management than a "way to help" it 
And if indeed uses the mmap() and munmap() OS syscalls directly to allocate and free that memory, then at least on *ix systems the only way I can see for it not to help with the Go memory management, is for the operating system virtual memory implementation to be b0rked -- not likely at all -- or else for rclone itself not to be releasing memory, either by not calling munmap() as it should, or due to memory being eaten by other uses not related to transfer buffers.
Thanks for the report @Rootax, and for the confirmation @kapitainsky, much appreciated.
As they say, the proof is in the pudding. I will start using that option as soon as I can abort current operations on the mount and then restart it with these, will then come back afterwards to report how it went.
It is possible to crash Go if you get memory allocated by mmap mixed up with normal memory. Rclone goes to some trouble to avoid this (by only using mmap memory for temporary buffers) and as far as I know it does, but that is the reason that it isn't the default.
No one has reported any problems using --use-mmap so maybe it should be the default! Though Go's memory allocator keeps improving.
Thanks for the very complete explanation, @ncw! Everything makes sense now.
I will make some more tests (my rclone mount keep being OOM-killed 
and will let you know of anything else I find.
This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.