Rclone mount crashing from exhausting memory (linux version 1.51.0)

Rclone mount crashing from exhausting memory on shared host and crashing:
Version:rclone v1.51.0

  • os/arch: linux/amd64
  • go version: go1.13.

Mounted using:
ExecStart=/home/rsb/bin/rclone mount pdrive2: /home/rsb/remgdrive
--allow-other
--config /home/rsb/.rclone.conf
--buffer-size 256M
--drive-chunk-size 32M
--dir-cache-time 96h
--timeout 1h
--umask 002
--vfs-cache-mode writes
--vfs-read-chunk-size 128M
--vfs-read-chunk-size-limit off

Error:
2020/02/03 23:52:49 Fatal error: Can not open: /home/rsb/remgdrive: open /home/rsb/remgdrive: transport endpoint is not connected
runtime: out of memory: cannot allocate 1048576-byte block (9796419584 in use)
fatal error: out of memory

runtime stack:
runtime.throw(0x16c9596, 0xd)
/opt/hostedtoolcache/go/1.13.7/x64/src/runtime/panic.go:774 +0x72
runtime.largeAlloc(0x100000, 0x460101, 0xc247e1c000)
/opt/hostedtoolcache/go/1.13.7/x64/src/runtime/malloc.go:1140 +0x16a
runtime.mallocgc.func1()
/opt/hostedtoolcache/go/1.13.7/x64/src/runtime/malloc.go:1033 +0x46
runtime.systemstack(0xc000281500)
/opt/hostedtoolcache/go/1.13.7/x64/src/runtime/asm_amd64.s:370 +0x66
runtime.mstart()
/opt/hostedtoolcache/go/1.13.7/x64/src/runtime/proc.go:1146

Deleted my previous stupid question, you are on golfing 1.13.7
How much RAM is on this system?

You have the buffer size too high as it while use 256M per file opened. You can remove that.

OP, you have 38 files open?
I believe there is some output missing from the log.

@Animosity022 That buffer should flush any time OP closes the buffered file. I think in order to quit with this error, OP would have to exceed Go's hard limit of 512GB; if OP is below that, I think the OS should just move it to swap, and keep running as normal? That's not ideal performance-wise, but shouldn't be fatal...

I'm not saying that removing the --buffer-size won't fix his/her issue, but would a 256GB buffer cause this on its own? Curious for my own education, as well as determining exactly what OP's error is

Just an update, removed buffer-size 256m line that was referenced in animosity repo rclone.service. So far it seems to be holding stable at 4 days in.

1 Like