I’m hoping I can get some help getting my memory usage under control. Over the course of a day my 64GB system gets to the point now where things start running out of memory, and my rclone usage at the time of writing has hit 50%.
I’m hoping I can get some tips on how to control my usage e.g. I don’t really understand what the checkers are doing in rclone move.
At the moment my setup is a a gdrive vfs mount as well as a new teamdrive mount (I intend to move all my gdrive files over to the teamdrive, so in time I will have one mount), at the time of posting only one Plex user active so quiet and 3 upload scripts (I use the teamdrive to do multiple user uploads).
rclone move /mnt/disk4/rclone_upload/google_vfs/ gdrive_media_vfs: -vv --drive-chunk-size 512M --checkers 3 --fast-list --transfers 3 --exclude .unionfs/** --exclude *fuse_hidden* --exclude *_HIDDEN --exclude .recycle** --exclude *.backup~* --exclude *.partial~* --delete-empty-src-dirs --fast-list --bwlimit 10000k --tpslimit 3 --min-age 30m
I suspect the multiple upload scripts are the problem. The teamdrive is a new addition to my setup today, but even before I was having high memory usage from normally 2 upload scripts running. I’m hoping I can solve as ideally I’d like to add a 4th upload/user to my teamdrive, so I can start backing up my server to gdrive via the mount.
Is there anything that jumps out in my upload command above or my mount below (I don’t think this is the problem as I only had one stream) that will help reduce memory usage, but maintain speed?
Thanks in advance
rclone mount --rc --allow-other --buffer-size 1G --dir-cache-time 72h --drive-chunk-size 256M --fast-list --log-level INFO --vfs-read-chunk-size 128M --vfs-read-chunk-size-limit off gdrive_media_vfs: /mnt/user/mount_rclone/google_vfs &
rclone mount --allow-other --buffer-size 1G --dir-cache-time 72h --drive-chunk-size 256M --fast-list --log-level INFO --vfs-read-chunk-size 128M --vfs-read-chunk-size-limit off tdrive1_media_vfs: /mnt/user/mount_rclone/google_tdrive1_vfs &
rclone rc --timeout=1h vfs/refresh recursive=true
Top detail
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
91463 root 20 0 9248304 8.1g 20312 S 205.6 12.8 78:18.61 qemu-system-x86
13168 root 20 0 4916404 4.2g 21980 S 420.4 6.7 4302:02 qemu-system-x86
16034 root 22 2 4240240 2.7g 18880 S 0.0 4.3 153:53.73 rcloneorig
92442 root 22 2 2297028 2.1g 18620 S 1.2 3.4 1:04.59 rcloneorig
125638 nobody 20 0 3451336 1.8g 12524 S 147.5 2.9 323:48.48 mono
2395 root 22 2 2907300 1.7g 18424 S 0.6 2.7 103:02.23 rcloneorig
84664 root 22 2 2840420 1.7g 18364 S 0.6 2.6 13:54.88 rcloneorig
43792 root 22 2 2837444 1.7g 17792 S 1.2 2.6 2:11.77 rcloneorig
90118 root 22 2 2905348 1.7g 18112 S 1.2 2.6 2:14.44 rcloneorig
130979 root 22 2 2905092 1.7g 18684 S 1.2 2.6 1:55.22 rcloneorig
58699 root 22 2 2837188 1.7g 18044 S 1.2 2.6 1:53.90 rcloneorig
7724 root 22 2 2363652 1.6g 18812 S 1.9 2.6 18:47.10 rcloneorig
44397 root 22 2 2363652 1.6g 18940 S 1.2 2.6 11:39.02 rcloneorig
41273 root 22 2 2296004 1.6g 17852 S 0.6 2.6 2:00.57 rcloneorig
49621 root 22 2 2296004 1.6g 18940 S 0.6 2.6 1:36.67 rcloneorig
501 root 22 2 2363652 1.6g 18428 S 1.2 2.6 1:54.58 rcloneorig
112654 root 22 2 2363652 1.6g 19196 S 1.2 2.6 10:43.81 rcloneorig
119946 root 22 2 2296004 1.6g 18496 S 0.6 2.6 1:49.07 rcloneorig
51280 root 22 2 2363908 1.6g 18236 S 0.6 2.6 1:59.20 rcloneorig
104066 root 22 2 2296004 1.6g 17404 S 1.2 2.6 1:56.62 rcloneorig
52449 root 22 2 1825444 1.6g 18684 S 1.2 2.6 19:00.34 rcloneorig
51766 root 22 2 1754564 1.6g 18236 S 1.2 2.6 0:28.51 rcloneorig
126861 nobody 20 0 4135984 1.2g 6212 S 0.6 1.9 35:27.94 mono
60172 root 22 2 2837444 1.1g 18816 S 0.6 1.8 11:01.22 rcloneorig
75467 root 22 2 2837700 1.1g 18876 S 0.6 1.8 14:12.75 rcloneorig
6466 root 22 2 2837956 1.1g 18876 S 0.6 1.8 8:55.87 rcloneorig
18561 nobody 20 0 19.2g 491444 0 S 0.6 0.7 6:23.63 java
17276 nobody 20 0 919988 447028 1416 S 18.5 0.7 218:30.36 transmission-da
15078 nobody 20 0 2629196 397996 0 S 0.0 0.6 44:58.63 mysqld
15657 nobody 20 0 5073588 392336 3036 S 19.1 0.6 180:13.29 nzbget
32966 nobody 20 0 5513528 350220 0 S 0.0 0.5 7:51.24 java
21736 root 20 0 5803832 336244 0 S 0.0 0.5 1:33.41 java
79930 root 20 0 12.7g 250636 3896 S 0.6 0.4 13:45.65 python
125352 nobody 20 0 1879972 215272 14876 S 0.6 0.3 298:29.64 Plex Media Serv
94936 root 22 2 3621836 198516 18368 S 0.0 0.3 191:36.04 rcloneorig
7410 root 20 0 4605636 196096 864 S 21.0 0.3 314:02.69 shfs
75251 nobody 20 0 1709500 178444 4712 S 0.0 0.3 0:10.88 mono
16632 nobody 20 0 306112 155516 224 S 0.0 0.2 15:04.26 python
126573 nobody 35 15 1715312 133908 0 S 0.0 0.2 2:39.00 Plex Script Hos
21817 nobody 20 0 3663244 132024 4544 S 0.6 0.2 5:42.93 calibre
122755 nobody 20 0 5226748 121816 9764 S 0.0 0.2 42:52.57 Ombi
20627 nobody 20 0 915916 107068 3884 S 1.2 0.2 7:27.59 mongod
8111 root 20 0 3591104 93688 34768 S 2.5 0.1 50:32.38 dockerd
16596 root 22 2 1688516 84812 18364 S 0.0 0.1 31:45.82 rcloneorig
124049 nobody 20 0 292888 74040 1880 S 0.6 0.1 5:45.14 python
17887 nobody 20 0 329992 71256 0 S 0.6 0.1 4:44.00 tvheadend
10623 nobody 20 0 229176 70320 0 S 0.0 0.1 0:22.46 python
8133 root 20 0 3208544 43004 19760 S 3.1 0.1 68:37.04 docker-containe
75231 nobody 20 0 95136 38988 5516 S 0.0 0.1 0:02.51 php-fpm7
56829 nobody 20 0 900316 38960 104 S 0.0 0.1 0:04.16 Plex Script Hos
121503 nobody 20 0 578512 37856 25140 S 0.0 0.1 0:06.69 php-fpm7
59863 nobody 20 0 889612 36512 0 S 0.0 0.1 0:06.09 Plex Script Hos
83368 nobody 20 0 576360 33312 22212 S 0.0 0.1 0:00.07 php-fpm7
59986 nobody 20 0 1769408 32388 0 S 0.0 0.0 0:04.26 Plex Script Hos
83400 nobody 20 0 575908 31556 21152 S 0.0 0.0 0:00.25 php-fpm7
16911 root 20 0 270664 30676 0 S 0.0 0.0 0:14.28 node
103081 root 22 2 132036 29900 18172 S 0.0 0.0 19:17.59 rcloneorig
110928 root 22 2 132292 29652 18300 S 0.0 0.0 26:02.51 rcloneorig