Help With High Memory Usage

I’m hoping I can get some help getting my memory usage under control. Over the course of a day my 64GB system gets to the point now where things start running out of memory, and my rclone usage at the time of writing has hit 50%.

I’m hoping I can get some tips on how to control my usage e.g. I don’t really understand what the checkers are doing in rclone move.

At the moment my setup is a a gdrive vfs mount as well as a new teamdrive mount (I intend to move all my gdrive files over to the teamdrive, so in time I will have one mount), at the time of posting only one Plex user active so quiet and 3 upload scripts (I use the teamdrive to do multiple user uploads).

rclone move /mnt/disk4/rclone_upload/google_vfs/ gdrive_media_vfs: -vv --drive-chunk-size 512M --checkers 3 --fast-list --transfers 3 --exclude .unionfs/** --exclude *fuse_hidden* --exclude *_HIDDEN --exclude .recycle** --exclude *.backup~* --exclude *.partial~* --delete-empty-src-dirs --fast-list --bwlimit 10000k --tpslimit 3 --min-age 30m

I suspect the multiple upload scripts are the problem. The teamdrive is a new addition to my setup today, but even before I was having high memory usage from normally 2 upload scripts running. I’m hoping I can solve as ideally I’d like to add a 4th upload/user to my teamdrive, so I can start backing up my server to gdrive via the mount.

Is there anything that jumps out in my upload command above or my mount below (I don’t think this is the problem as I only had one stream) that will help reduce memory usage, but maintain speed?

Thanks in advance

rclone mount --rc --allow-other --buffer-size 1G --dir-cache-time 72h --drive-chunk-size 256M --fast-list --log-level INFO --vfs-read-chunk-size 128M --vfs-read-chunk-size-limit off gdrive_media_vfs: /mnt/user/mount_rclone/google_vfs &

rclone mount --allow-other --buffer-size 1G --dir-cache-time 72h --drive-chunk-size 256M --fast-list --log-level INFO --vfs-read-chunk-size 128M --vfs-read-chunk-size-limit off tdrive1_media_vfs: /mnt/user/mount_rclone/google_tdrive1_vfs &

rclone rc --timeout=1h vfs/refresh recursive=true

Top detail

   PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
 91463 root      20   0 9248304   8.1g  20312 S 205.6  12.8  78:18.61 qemu-system-x86
 13168 root      20   0 4916404   4.2g  21980 S 420.4   6.7   4302:02 qemu-system-x86
 16034 root      22   2 4240240   2.7g  18880 S   0.0   4.3 153:53.73 rcloneorig
 92442 root      22   2 2297028   2.1g  18620 S   1.2   3.4   1:04.59 rcloneorig
125638 nobody    20   0 3451336   1.8g  12524 S 147.5   2.9 323:48.48 mono
  2395 root      22   2 2907300   1.7g  18424 S   0.6   2.7 103:02.23 rcloneorig
 84664 root      22   2 2840420   1.7g  18364 S   0.6   2.6  13:54.88 rcloneorig
 43792 root      22   2 2837444   1.7g  17792 S   1.2   2.6   2:11.77 rcloneorig
 90118 root      22   2 2905348   1.7g  18112 S   1.2   2.6   2:14.44 rcloneorig
130979 root      22   2 2905092   1.7g  18684 S   1.2   2.6   1:55.22 rcloneorig
 58699 root      22   2 2837188   1.7g  18044 S   1.2   2.6   1:53.90 rcloneorig
  7724 root      22   2 2363652   1.6g  18812 S   1.9   2.6  18:47.10 rcloneorig
 44397 root      22   2 2363652   1.6g  18940 S   1.2   2.6  11:39.02 rcloneorig
 41273 root      22   2 2296004   1.6g  17852 S   0.6   2.6   2:00.57 rcloneorig
 49621 root      22   2 2296004   1.6g  18940 S   0.6   2.6   1:36.67 rcloneorig
   501 root      22   2 2363652   1.6g  18428 S   1.2   2.6   1:54.58 rcloneorig
112654 root      22   2 2363652   1.6g  19196 S   1.2   2.6  10:43.81 rcloneorig
119946 root      22   2 2296004   1.6g  18496 S   0.6   2.6   1:49.07 rcloneorig
 51280 root      22   2 2363908   1.6g  18236 S   0.6   2.6   1:59.20 rcloneorig
104066 root      22   2 2296004   1.6g  17404 S   1.2   2.6   1:56.62 rcloneorig
 52449 root      22   2 1825444   1.6g  18684 S   1.2   2.6  19:00.34 rcloneorig
 51766 root      22   2 1754564   1.6g  18236 S   1.2   2.6   0:28.51 rcloneorig
126861 nobody    20   0 4135984   1.2g   6212 S   0.6   1.9  35:27.94 mono
 60172 root      22   2 2837444   1.1g  18816 S   0.6   1.8  11:01.22 rcloneorig
 75467 root      22   2 2837700   1.1g  18876 S   0.6   1.8  14:12.75 rcloneorig
  6466 root      22   2 2837956   1.1g  18876 S   0.6   1.8   8:55.87 rcloneorig
 18561 nobody    20   0   19.2g 491444      0 S   0.6   0.7   6:23.63 java
 17276 nobody    20   0  919988 447028   1416 S  18.5   0.7 218:30.36 transmission-da
 15078 nobody    20   0 2629196 397996      0 S   0.0   0.6  44:58.63 mysqld
 15657 nobody    20   0 5073588 392336   3036 S  19.1   0.6 180:13.29 nzbget
 32966 nobody    20   0 5513528 350220      0 S   0.0   0.5   7:51.24 java
 21736 root      20   0 5803832 336244      0 S   0.0   0.5   1:33.41 java
 79930 root      20   0   12.7g 250636   3896 S   0.6   0.4  13:45.65 python
125352 nobody    20   0 1879972 215272  14876 S   0.6   0.3 298:29.64 Plex Media Serv
 94936 root      22   2 3621836 198516  18368 S   0.0   0.3 191:36.04 rcloneorig
  7410 root      20   0 4605636 196096    864 S  21.0   0.3 314:02.69 shfs
 75251 nobody    20   0 1709500 178444   4712 S   0.0   0.3   0:10.88 mono
 16632 nobody    20   0  306112 155516    224 S   0.0   0.2  15:04.26 python
126573 nobody    35  15 1715312 133908      0 S   0.0   0.2   2:39.00 Plex Script Hos
 21817 nobody    20   0 3663244 132024   4544 S   0.6   0.2   5:42.93 calibre
122755 nobody    20   0 5226748 121816   9764 S   0.0   0.2  42:52.57 Ombi
 20627 nobody    20   0  915916 107068   3884 S   1.2   0.2   7:27.59 mongod
  8111 root      20   0 3591104  93688  34768 S   2.5   0.1  50:32.38 dockerd
 16596 root      22   2 1688516  84812  18364 S   0.0   0.1  31:45.82 rcloneorig
124049 nobody    20   0  292888  74040   1880 S   0.6   0.1   5:45.14 python
 17887 nobody    20   0  329992  71256      0 S   0.6   0.1   4:44.00 tvheadend
 10623 nobody    20   0  229176  70320      0 S   0.0   0.1   0:22.46 python
  8133 root      20   0 3208544  43004  19760 S   3.1   0.1  68:37.04 docker-containe
 75231 nobody    20   0   95136  38988   5516 S   0.0   0.1   0:02.51 php-fpm7
 56829 nobody    20   0  900316  38960    104 S   0.0   0.1   0:04.16 Plex Script Hos
121503 nobody    20   0  578512  37856  25140 S   0.0   0.1   0:06.69 php-fpm7
 59863 nobody    20   0  889612  36512      0 S   0.0   0.1   0:06.09 Plex Script Hos
 83368 nobody    20   0  576360  33312  22212 S   0.0   0.1   0:00.07 php-fpm7
 59986 nobody    20   0 1769408  32388      0 S   0.0   0.0   0:04.26 Plex Script Hos
 83400 nobody    20   0  575908  31556  21152 S   0.0   0.0   0:00.25 php-fpm7
 16911 root      20   0  270664  30676      0 S   0.0   0.0   0:14.28 node
103081 root      22   2  132036  29900  18172 S   0.0   0.0  19:17.59 rcloneorig
110928 root      22   2  132292  29652  18300 S   0.0   0.0  26:02.51 rcloneorig

I’m not sure why you’d see a lot of memory usage from the upload scripts as those are you the default buffer size.

The mount command is using a 1G buffer so depending on the files that are opened, that would consume a lot of memory.

Here’s my example of memory in use:

felix@gemini:~$ ps aux --sort -rss | head
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
felix     7228  3.9 12.8 6143796 4219864 ?     Sl   Nov27 1108:46 /usr/bin/mono --debug /opt/NzbDrone/NzbDrone.exe /data=/opt/sonarrdata /nobrowser /restart
felix     7337  0.5 11.3 5182368 3723572 ?     Ssl  Nov27 154:54 /usr/bin/mono /opt/Radarr/Radarr.exe --nobrowser --data=/opt/radarrdata
unifi     3065  0.1  2.3 1071144 768860 ?      Sl   Nov27  53:28 bin/mongod --dbpath /usr/lib/unifi/data/db --port 27117 --unixSocketPrefix /usr/lib/unifi/run --logappend --logpath /usr/lib/unifi/logs/mongod.log --nohttpinterface --bind_ip 127.0.0.1
unifi     1653  0.1  2.3 7613700 768584 ?      Sl   Nov27  44:18 /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -Xmx1024M -XX:ErrorFile=/usr/lib/unifi/logs/hs_err_pid%p.log -Dapple.awt.UIElement=true -jar /usr/lib/unifi/lib/ace.jar start
influxdb  1025  1.0  1.7 8401352 561212 ?      Ssl  Nov27 298:56 /usr/bin/influxd -config /etc/influxdb/influxdb.conf
felix     1053  1.1  1.3 1943180 429444 ?      Sl   Nov27 328:13 /usr/bin/python -u /opt/bazarr/bazarr/main.py
felix    16207  0.7  0.9 1694608 317812 ?      Sl   Dec15  16:24 Plex Plug-in [com.plexapp.plugins.trakttv] /usr/lib/plexmediaserver/Resources/Plug-ins-dcc512d26/Framework.bundle/Contents/Resources/Versions/2/Python/bootstrap.py --server-version 1.14.1.5487-dcc512d26 /var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-ins/Trakttv.bundle
felix    15906  0.2  0.7 3594232 236956 ?      Sl   Dec01  51:23 /usr/bin/python /opt/Tautulli/Tautulli.py --quiet --daemon --nolaunch --config /opt/Tautulli/config.ini --datadir /opt/Tautulli
felix    17443  0.7  0.7 1827912 233836 ?      Ssl  Dec05 117:01 /usr/bin/rclone mount gcrypt: /GD --allow-other --bind 192.168.1.30 --buffer-size 256M --dir-cache-time 72h --drive-chunk-size 32M --log-level INFO --log-file /home/felix/logs/rclone.log --umask 002 --vfs-read-chunk-size 128M --vfs-read-chunk-size-limit off --rc

You really want to hone in the %MEM as VIRT is misleading.

I don’t think the 1G is affecting the upload scripts and I only had one stream open when I grabbed top. I’m going to switch to your 256M anyway to see if that helps.

I’m wondering now if it’s plex or one radarr or sonarr that’s driving the high usage and not the upload scripts, when they are analysing files? When sonarr etc look at files I’m guessing the 1G buffer comes into play?

Why are you running so many rclones?

are they not each upload instance i.e. each file uploading? If not, that might be what’s causing the problem. I’ve been experimenting with teamdrive today i.e. I’ve been mounting and re-mounting a lot. But, each time I unmounted with:

fusermount -uz /mount_location

so, wouldn’t this kill the old mount? I’m giong to do a reboot now and run for a few hours and see if all the multiple instances disappear

Why do you run 3 transfers per upload mount? I think that will increase your memory usage exponentially for each mount uploading. And since your upload is maxxed by bwlimit anyways my reasoning is to just run 1 transfer to keep the transfer time per file as low as possible.

Regardless of how many checkers/transfers/etc you run, there should only be one process for each rclone copy or move command running.

You have quite the number of our rclone processes running, which I am not sure why there are that many.

the fusermount unmounts the location and stops the rclone mount process.

I read somewhere that more than one transfer is needed to hit the bwlimit.

I’ve just rebooted and I’m seeing 6 rclone instances which makes sense - 2 mounts and 4 uploads. I think the mess before was because of all my experiments with teamdrives.

I’m going to leave it alone now and see how it gets on for a day or two.

It sounds like you had a script or something that was causing your duplicate uploads going on.

yep that was exactly it :frowning:

Total usage now is about 10GB for 2 mounts and 4 upload scripts (not including streams) running at a total of about 250MB/s. I can live with that as I won’t be uploading at this speed constantly

1 Like