Out of memory: Kill process xxxx (rclone) score 972 or sacrifice child

Help and Support
1

@ncw

Hi mate.

My server crashed the other day , and I had to reinstall my entire Ubuntu 16.04 server from scratch. I do as I normally do - follow my own steps to get everything up and running. Server has 64GB memory and 2 x 512GB ssd drives in raid 0. AMD Ryzen 1700x

So the server is running, but it seems that my plexserver loose its Googledrive mount connections which I never had the problem with before.

I opened up my kern.log and I scroll through it, and it seems that my server kills the rclone process?

ex:

Out of memory: Kill process 2609 (rclone) score 972 or sacrifice child
Killed process 2609 (rclone) total-vm: 74290364kb, aron-rss:64081484kB, file-rss:0kb, shmem-rss:0kb
oom_reaper : reaped process 2609 (rclone) now aron-rss:0kb , file-rss:0kb, shmem-rss:0kb

Any clue?

Or what can I do to get details logs if my server looses connections to the mount again?

Thanks

Morphy

2

refer this link to add DEBUG log

https://rclone.org/docs/#log-file-file

3

Thanks

Should I add the line to my mount:

–log-file /home/myuser/logs/rclone.log -vv ?

4

The kernel killed the rclone process, because it was running out of memory and rclone was using nearly 64 GB. Can you give us some information about your rclone setup, like the relevant parts of your rclone.conf and the used mount command?

There is a pending issue on GitHub regarding rclone using too much memory, but there is progress at the moment, since it only seem to happen very rarely and is not reproducible.
If this happens again (and you notice it before the kernel kills rclone) could you take a memory profile of your rclone process?
It should pretty simple to setup: install go, enable --rc for your rclone mount and when the memory usage is to high (maybe > 5 GB) use
go tool pprof -web http://localhost:5572/debug/pprof/heap
to get a memory profile.

5

Use
–rc --log-file /home/myuser/logs/rclone.log --log-level DEBUG

6

Thanks for reply to you all

Running Rclone version 1.43.1 on Ubuntu 16.04.5 LTS

My systemd file

[Unit]
Description=RClone Service
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
ExecStart=/usr/bin/rclone mount --allow-other --read-only --dir-cache-time 48h --vfs-read-chunk-size 16M --rc --log-file /home/plex/logs/rclone.log --log-level DEBUG --buffer-size 2G Googlecrypt: /home/plex/mnt
ExecStop=/bin/fusermount -uz /home/plex/mnt
Restart=on-abort
User=plex
Group=plex

[Install]
WantedBy=default.target

I will see if I can look into your install go thingy , but I dont quite monitor my server all the time. I just hear that my plex users complains when the server dont have access to my media.
Can I do a “roll back” to another version that shouldnt cause this? If yes which version it 100% sure to run?

Added: --rc --log-file /home/plex/logs/rclone.log --log-level DEBUG

Thanks :slight_smile:

7

I am using latest version plex media server (Version 1.13.8.5395) on my NAT OpenVZ VPS with 1 GB Ram, no issue until today.

I hope it can fix your rclone mount trouble.

8

And here is my rclone.conf

[Google]
type = drive
client_id = xxxxxxx
client_secret = xxxxx
service_account_file =
token = {“access_token”:"xxxx

[Googlecrypt]
type = crypt
remote = Google:Private
filename_encryption = standard
directory_name_encryption = true
password = xxxxxxxxx
password2 =

9

buffer-size is per “open file” , please remember that. 10 open files/streams => 20GB RAM usage

10

Yes im aware . So If i have ex 10 users = 20GB as they stream correct ?

11

But as I wrote earlier - Never encountered this issue before intollerant now .

12

from my own experience you will likely have the same performance even with buffer-size=16M

13

Im a bit confused about the buffer-size , what would be the recommended setting for this one?

14

The default is 16M I believe. It’s all about how many open files you’ll have and how much memory you have the system.

15

Hi Ani :slight_smile:

Well I followed your guide , I see you have 2GB in your settings.

I have 64GB memory - what would u prefer mate?

Lets say my max streams is aprox 10-11 users at the time, what should I go for?

16

So, it depends.

My library is fully analyzed so I use 2GB for now as I direct play a lot and like the buffer.

I only have 3-4 streams going at a time, 32GB of memory and a larger swap in case of something awful.

If you are initializing a library, a much lower value is needed as it opens 30-50 files at a time.

My advice is to use the defaults and go larger if needed based on your results.

17

The server still shuts down rclone since its out of memory .

Dont got any clue what so ever.

18

Ok server keeps losing connection - Im just for the “fun” of it trying to use Plexdrive to see how the system is acting.

19

Unfortunately, we don’t either as there isn’t much in terms of logs or anything else to look at.

If you are running out of memory, it’s most likely due to the number of open files by whatever is accessing the mount. You can run (as root) lsof on the mount and see what’s accessing files.

Or you can follow the post from a few above where he asked to grab a memory profile.

20

Yea but the logs are huuuuuge Ani - The only thing I can see is that the kern.log is killing rclone every time .
What in the world that is causing it, dont know ! Its sooo damn odd since I ran my reinstall , made my services which I have a copy of. Everything is made as it was before. Haven’t done any changes at all. But after the server reinstallation , it keep eating the memory.

Now im running Plexdrive which seems to be working so far , so good for now. Memory usage is insane low and not moving . 662MB used while I stream a movie direct play so far. Starting time is aprox … maybe a bit slower than rclone vfs .