Why is rclone using swap?

Help and Support
1

I have been setting up a new server noticed that rclone seems to be using swap even when there is a ton of memory available.

rclone v1.42
- os/arch: linux/amd64
- go version: go1.10.1

ExecStart=/usr/bin/rclone mount gcrypt: /home/admin/mnt/gdrive \
   --allow-other \
   --tpslimit 4 \
   --dir-cache-time 72h \
   --buffer-size 512M \
   --vfs-read-chunk-size 64M \
   --vfs-read-chunk-size-limit 1G \
   --umask 002 \
   --bind MYIP \
   --bwlimit 30M \
   --log-level INFO \

The system has 32GB of RAM and 1GB of swap.

RAM utilization is around 10%.

vm.swappiness is set to 0.

Right now Rclone is using around 60MB of swap which isn’t much but I would still like to know why.

It seems to spike up when I am downloading something to a local drive which is merged with my rclone mount using unionfs. I do not have any kind of syncing to G-Drive for locally downloaded files.

Any ideas?

Should I just turn off swap entirely?

Edit: I just downloaded another file and rclone swap usage just doubled to 120MB.

2

What distro are you running?

There are quite a few settings that impact what is being swapped in / out. Ubuntu likes to have a lot of things swap out where default Debian does a good job of not using any swap.

3

I am running Ubuntu 16.04.

4

So for what it’s worth, these are the defaults that are on Debian 9.5

root@gemini:~ sysctl -a  | grep ^vm
vm.admin_reserve_kbytes = 8192
vm.block_dump = 0
vm.compact_unevictable_allowed = 1
vm.dirty_background_bytes = 0
vm.dirty_background_ratio = 10
vm.dirty_bytes = 0
vm.dirty_expire_centisecs = 3000
vm.dirty_ratio = 20
vm.dirty_writeback_centisecs = 500
vm.dirtytime_expire_seconds = 43200
vm.drop_caches = 0
vm.extfrag_threshold = 500
vm.hugepages_treat_as_movable = 0
vm.hugetlb_shm_group = 0
vm.laptop_mode = 0
vm.legacy_va_layout = 0
vm.lowmem_reserve_ratio = 256	256	32	1
vm.max_map_count = 65530
vm.memory_failure_early_kill = 0
vm.memory_failure_recovery = 1
vm.min_free_kbytes = 67584
vm.min_slab_ratio = 5
vm.min_unmapped_ratio = 1
vm.mmap_min_addr = 65536
vm.mmap_rnd_bits = 28
vm.mmap_rnd_compat_bits = 8
vm.nr_hugepages = 0
vm.nr_hugepages_mempolicy = 0
vm.nr_overcommit_hugepages = 0
vm.nr_pdflush_threads = 0
vm.numa_zonelist_order = default
vm.oom_dump_tasks = 1
vm.oom_kill_allocating_task = 0
vm.overcommit_kbytes = 0
vm.overcommit_memory = 0
vm.overcommit_ratio = 50
vm.page-cluster = 3
vm.panic_on_oom = 0
vm.percpu_pagelist_fraction = 0
vm.stat_interval = 1
vm.swappiness = 60
vm.user_reserve_kbytes = 131072
vm.vfs_cache_pressure = 100
vm.watermark_scale_factor = 10
vm.zone_reclaim_mode = 0

and my free:

root@gemini:~ free
              total        used        free      shared  buff/cache   available
Mem:       32847364     3589820      520876       32512    28736668    28757088
Swap:      33449980           0    33449980
5

This means every open file can potentially use 512M of memory. rclone will allocate that memory then free it back to the OS if it can, however that isn’t always 100% successful.

So I’d recommend reducing that.

You could also try this experimental rclone which uses mmap for buffer memory allocation which is 100% at giving the memory back to the OS.

https://beta.rclone.org/branch/v1.43-007-g9f425414-mmap-beta/ (uploaded in 15-30 mins)

6

I get 0MB swap usage even with big buffers. I have 32GB of memory and I never compared, but there is a vm setting(s) that are different that cause Debain to never swap out on me, but the same config for rclone swaps out a little on Ubuntu (I was using 18.04).

Quick pic from playing some items this morning where my buffer jumps up, but graph on the right shows my swap at all free.

image
image.png2598×1290 240 KB

and my free:

root@gemini:~ free
              total        used        free      shared  buff/cache   available
Mem:       32847364     3977600     2439816       32512    26429948    28369592
Swap:      33449980           0    33449980
7

So here is the outout from the same command:

vm.admin_reserve_kbytes = 8192
vm.block_dump = 0
vm.compact_unevictable_allowed = 1
vm.dirty_background_bytes = 0
vm.dirty_background_ratio = 10
vm.dirty_bytes = 0
vm.dirty_expire_centisecs = 3000
vm.dirty_ratio = 20
vm.dirty_writeback_centisecs = 500
vm.dirtytime_expire_seconds = 43200
vm.drop_caches = 0
vm.extfrag_threshold = 500
vm.hugepages_treat_as_movable = 0
vm.hugetlb_shm_group = 0
vm.laptop_mode = 0
vm.legacy_va_layout = 0
vm.lowmem_reserve_ratio = 256   256     32      1
vm.max_map_count = 65530
vm.memory_failure_early_kill = 0
vm.memory_failure_recovery = 1
vm.min_free_kbytes = 90112
vm.min_slab_ratio = 5
vm.min_unmapped_ratio = 1
vm.mmap_min_addr = 65536
vm.nr_hugepages = 0
vm.nr_hugepages_mempolicy = 0
vm.nr_overcommit_hugepages = 0
vm.nr_pdflush_threads = 0
vm.numa_zonelist_order = default
vm.oom_dump_tasks = 1
vm.oom_kill_allocating_task = 0
vm.overcommit_kbytes = 0
vm.overcommit_memory = 1
vm.overcommit_ratio = 50
vm.page-cluster = 3
vm.panic_on_oom = 0
vm.percpu_pagelist_fraction = 0
vm.stat_interval = 1
vm.swappiness = 0
vm.user_reserve_kbytes = 131072
vm.vfs_cache_pressure = 100
vm.zone_reclaim_mode = 0

I just upgraded to the latest sable release of rclone and it seems to be behaving better with most swap usage now coming from Docker although rclone is still using some.

@ncw Thanks for the tip and I might give it a try but my understanding is that should still be all in RAM. I see mem usage jump when I start a stream which I assume is related to this but it doesn’t seem to have any effect on SWAP.

8

After comparing out vm settings I changed vm.overcommit_memory from 1 to 0 and will see if that changes anything.

9

I’ve tried a bit tweaking on Ubuntu and never got into the situation of using 0 swap. On a stock debian install, it never uses swap.

At some point, I’ll spend some time to figure out what is different.

10

Still having issues. rclone is now using 800MB of swap with 4 video streams running. Last night I has similar usage but no increasing swap utilization.

11

What’s the actual issue you are seeing? Just swap use? Is there something else happening?

12

No actual performance issues that I can see. I am just at around 80% swap use currently. I will update if I notice any issues pop up once I am out of swap space. I still have 75% free ram.

13

I couldn’t figure it out on Ubuntu as to why it was using swap.

I read through a bunch of things with this one being the most helpful although it didn’t solve my swap question either.

https://help.ubuntu.com/community/SwapFaq

I didn’t find it to be rclone, but anything that used a lot of memory so Sonarr/Radarr/Ombi would all swap out some even with swappiness of 0.

I run with the default settings on Debian 9 and get 0 swap use ever. I can’t figure out what else is different that makes the swap use so different:

felix@gemini:~$ cat /proc/sys/vm/swappiness
60
felix@gemini:~$ free
total used free shared buff/cache available
Mem: 32847364 7088620 331800 49072 25426944 25241604
Swap: 33449980 0 33449980
felix@gemini:~$

and I got ~3GB of memory in use with 5 streams going on:

image
image.png1300×130 67.7 KB

That’s where I get a bit confused as I can see the VIRT memory higher than RES but still no swap use. My unifi AP software is taking 7.4GB of VIRT but only 500M of RES?

14

Thanks I will give it a read. A quick update, I watched swap hit around 90% and then the os or rcone seemed to empty the swap and utilization dropped down to 7%.

I will read up and see if this is even an issue. Thanks again for all your help.

15

What does your free output look like?

16

rclone and the go runtime uses lots of virtual memory. That is where it has allocated memory but hasn’t used it yet, so the OS hasn’t actually supplied it.

With some settings of vm.overcommit_memory the linux kernel will back all virtual memory with real memory immediately. Check that with cat /proc/sys/vm/overcommit_memory. The default is 0.

17

0 on debian for me. I just never swap on Debian at all though. It’s odd:

      total        used        free      shared  buff/cache   available

Mem: 32077 2660 286 51 29130 28908
Swap: 32665 0 32665

18 
              total        used        free      shared  buff/cache   available
Mem:       32909996     2693600      277604       48764    29938792    29598988
Swap:       1141752      140516     1001236

cat /proc/sys/vm/overcommit_memory = 0

19

So this is a funny site that does a superb job at explaining memory and swap:

https://www.linuxatemyram.com/

What still doesn’t make sense to me is why it is using any swap. Based on all the sysctl settings and everything else, you have plenty of memory so swap should never be used. I don’t think it’s just rclone as you’ll find it is any process that consumes a lot of memory.

You can use something like this to see what’s actually using swap as mine turns 0 for everything:

for file in /proc/*/status ; do awk '/VmSwap|Name/{printf $2 " " $3}END{ print ""}' $file; done | sort -k 2 -n -r | less

20

Yes its not only rclone but it is using the vast majority. I have a bash script to show swap usage and this is the print out:

PID=1 swapped 28 KB (systemd)
PID=503 swapped 8 KB (systemd-journal)
PID=737 swapped 97068 KB (rclone)
PID=882 swapped 52 KB (systemd-timesyn)
PID=953 swapped 64 KB (rsyslogd)
PID=966 swapped 24 KB (accounts-daemon)
PID=973 swapped 12 KB (systemd-logind)
PID=1010 swapped 96 KB (monit)
PID=1017 swapped 208 KB (sshd)
PID=1033 swapped 14080 KB (dockerd)
PID=1053 swapped 16 KB (login)
PID=1082 swapped 8108 KB (docker-containe)
PID=1791 swapped 280 KB (docker-containe)
PID=2062 swapped 1280 KB (docker-proxy)
PID=2079 swapped 736 KB (docker-proxy)
PID=2144 swapped 80 KB (docker-containe)
PID=2248 swapped 140 KB (docker-containe)
PID=2944 swapped 160 KB (docker-containe)
PID=2984 swapped 188 KB (docker-containe)
PID=3079 swapped 716 KB (docker-proxy)
PID=3451 swapped 208 KB (docker-containe)
PID=4036 swapped 116 KB (docker-containe)
PID=6239 swapped 188 KB (docker-proxy)
PID=6252 swapped 8 KB (docker-proxy)
PID=6266 swapped 1200 KB (docker-proxy)
PID=6282 swapped 1188 KB (docker-proxy)
PID=6296 swapped 520 KB (docker-proxy)
PID=6309 swapped 1188 KB (docker-proxy)
PID=6323 swapped 1184 KB (docker-proxy)
PID=6336 swapped 1024 KB (docker-proxy)
PID=6350 swapped 1376 KB (docker-proxy)
PID=6363 swapped 1204 KB (docker-proxy)
PID=6379 swapped 1184 KB (docker-proxy)
PID=6386 swapped 80 KB (docker-containe)
PID=7241 swapped 160 KB (unbound)
PID=8805 swapped 120 KB (docker-containe)
PID=14580 swapped 152 KB (docker-containe)
PID=20346 swapped 124 KB (fail2ban-server)
PID=28328 swapped 504 KB (netdata)
PID=28360 swapped 5548 KB (python)
Overall swap used: 140620 KB

I am going to test enabling swap accounting with the by adding swapaccount=1 to the grub config and see if this has any effect.