Rclone mount crashing from exhausting memory (linux version 1.51.0)

#1

Rclone mount crashing from exhausting memory on shared host and crashing:
Version:rclone v1.51.0

  • os/arch: linux/amd64
  • go version: go1.13.

Mounted using:
ExecStart=/home/rsb/bin/rclone mount pdrive2: /home/rsb/remgdrive
--allow-other
--config /home/rsb/.rclone.conf
--buffer-size 256M
--drive-chunk-size 32M
--dir-cache-time 96h
--timeout 1h
--umask 002
--vfs-cache-mode writes
--vfs-read-chunk-size 128M
--vfs-read-chunk-size-limit off

Error:
2020/02/03 23:52:49 Fatal error: Can not open: /home/rsb/remgdrive: open /home/rsb/remgdrive: transport endpoint is not connected
runtime: out of memory: cannot allocate 1048576-byte block (9796419584 in use)
fatal error: out of memory

runtime stack:
runtime.throw(0x16c9596, 0xd)
/opt/hostedtoolcache/go/1.13.7/x64/src/runtime/panic.go:774 +0x72
runtime.largeAlloc(0x100000, 0x460101, 0xc247e1c000)
/opt/hostedtoolcache/go/1.13.7/x64/src/runtime/malloc.go:1140 +0x16a
runtime.mallocgc.func1()
/opt/hostedtoolcache/go/1.13.7/x64/src/runtime/malloc.go:1033 +0x46
runtime.systemstack(0xc000281500)
/opt/hostedtoolcache/go/1.13.7/x64/src/runtime/asm_amd64.s:370 +0x66
runtime.mstart()
/opt/hostedtoolcache/go/1.13.7/x64/src/runtime/proc.go:1146

#2

Deleted my previous stupid question, you are on golfing 1.13.7
How much RAM is on this system?

#3

You have the buffer size too high as it while use 256M per file opened. You can remove that.

#4

OP, you have 38 files open?
I believe there is some output missing from the log.

@Animosity022 That buffer should flush any time OP closes the buffered file. I think in order to quit with this error, OP would have to exceed Go's hard limit of 512GB; if OP is below that, I think the OS should just move it to swap, and keep running as normal? That's not ideal performance-wise, but shouldn't be fatal...

I'm not saying that removing the --buffer-size won't fix his/her issue, but would a 256GB buffer cause this on its own? Curious for my own education, as well as determining exactly what OP's error is

#5

Just an update, removed buffer-size 256m line that was referenced in animosity repo rclone.service. So far it seems to be holding stable at 4 days in.

1 Like