Right now Rclone is using around 60MB of swap which isn’t much but I would still like to know why.
It seems to spike up when I am downloading something to a local drive which is merged with my rclone mount using unionfs. I do not have any kind of syncing to G-Drive for locally downloaded files.
Any ideas?
Should I just turn off swap entirely?
Edit: I just downloaded another file and rclone swap usage just doubled to 120MB.
There are quite a few settings that impact what is being swapped in / out. Ubuntu likes to have a lot of things swap out where default Debian does a good job of not using any swap.
This means every open file can potentially use 512M of memory. rclone will allocate that memory then free it back to the OS if it can, however that isn’t always 100% successful.
So I’d recommend reducing that.
You could also try this experimental rclone which uses mmap for buffer memory allocation which is 100% at giving the memory back to the OS.
I get 0MB swap usage even with big buffers. I have 32GB of memory and I never compared, but there is a vm setting(s) that are different that cause Debain to never swap out on me, but the same config for rclone swaps out a little on Ubuntu (I was using 18.04).
Quick pic from playing some items this morning where my buffer jumps up, but graph on the right shows my swap at all free.
I just upgraded to the latest sable release of rclone and it seems to be behaving better with most swap usage now coming from Docker although rclone is still using some.
@ncw Thanks for the tip and I might give it a try but my understanding is that should still be all in RAM. I see mem usage jump when I start a stream which I assume is related to this but it doesn’t seem to have any effect on SWAP.
Still having issues. rclone is now using 800MB of swap with 4 video streams running. Last night I has similar usage but no increasing swap utilization.
No actual performance issues that I can see. I am just at around 80% swap use currently. I will update if I notice any issues pop up once I am out of swap space. I still have 75% free ram.
That’s where I get a bit confused as I can see the VIRT memory higher than RES but still no swap use. My unifi AP software is taking 7.4GB of VIRT but only 500M of RES?
Thanks I will give it a read. A quick update, I watched swap hit around 90% and then the os or rcone seemed to empty the swap and utilization dropped down to 7%.
I will read up and see if this is even an issue. Thanks again for all your help.
rclone and the go runtime uses lots of virtual memory. That is where it has allocated memory but hasn’t used it yet, so the OS hasn’t actually supplied it.
With some settings of vm.overcommit_memory the linux kernel will back all virtual memory with real memory immediately. Check that with cat /proc/sys/vm/overcommit_memory. The default is 0.
What still doesn’t make sense to me is why it is using any swap. Based on all the sysctl settings and everything else, you have plenty of memory so swap should never be used. I don’t think it’s just rclone as you’ll find it is any process that consumes a lot of memory.
You can use something like this to see what’s actually using swap as mine turns 0 for everything:
for file in /proc/*/status ; do awk '/VmSwap|Name/{printf $2 " " $3}END{ print ""}' $file; done | sort -k 2 -n -r | less