Rclone cryptcheck running out of memory

Suspected Bug
1

Hello,

So, due to my “rclone mount // md5sum -c” taking inordinate amounts of time in my ~8M files / ~3.5TB directory (see here for more details), I tried using rclone cryptcheck to check it against my local copy; unfortunately, it abended in less than 2 hours with an out-of-memory condition, while still “Building file list” for the whole tree:

fatal error: runtime: out of memory

runtime stack:
runtime.throw(0xe4d97c, 0x16)
        /opt/go/go1.8/src/runtime/panic.go:596 +0x95
runtime.sysMap(0xc65cc90000, 0x100000, 0xc4202bbd00, 0x1389458)
        /opt/go/go1.8/src/runtime/mem_linux.go:216 +0x1d0
runtime.(*mheap).sysAlloc(0x1370320, 0x100000, 0x7f4c2b3eea30)
        /opt/go/go1.8/src/runtime/malloc.go:440 +0x374
runtime.(*mheap).grow(0x1370320, 0x1, 0x0)
        /opt/go/go1.8/src/runtime/mheap.go:774 +0x62
runtime.(*mheap).allocSpanLocked(0x1370320, 0x1, 0xc4202bbe20)
        /opt/go/go1.8/src/runtime/mheap.go:678 +0x44f
runtime.(*mheap).alloc_m(0x1370320, 0x1, 0x3, 0x7f4c2b3eea78)
        /opt/go/go1.8/src/runtime/mheap.go:562 +0xe2
runtime.(*mheap).alloc.func1()
        /opt/go/go1.8/src/runtime/mheap.go:627 +0x4b
runtime.systemstack(0xc4202bbe90)
        /opt/go/go1.8/src/runtime/asm_amd64.s:343 +0xab
runtime.(*mheap).alloc(0x1370320, 0x1, 0x10000000003, 0x7f4c2b3eea30)
        /opt/go/go1.8/src/runtime/mheap.go:628 +0xa0
runtime.(*mcentral).grow(0x13717a0, 0x0)
        /opt/go/go1.8/src/runtime/mcentral.go:212 +0x8e
runtime.(*mcentral).cacheSpan(0x13717a0, 0xc42001b228)
        /opt/go/go1.8/src/runtime/mcentral.go:93 +0x12a
runtime.(*mcache).refill(0x7f4c3ec8e000, 0xc600000003, 0xc65cc8edf4)
        /opt/go/go1.8/src/runtime/mcache.go:122 +0xa0
runtime.(*mcache).nextFree.func1()
        /opt/go/go1.8/src/runtime/malloc.go:538 +0x32
runtime.systemstack(0xc42001a000)
        /opt/go/go1.8/src/runtime/asm_amd64.s:327 +0x79
runtime.mstart()
        /opt/go/go1.8/src/runtime/proc.go:1132

This is followed by ~13K lines of “goroutine” stack dumps.

Seems like that entire dirtree metadata will not fit in memory :-/

I remember having a similar issue a few months back with rclone sync, and that @ncw nixed it by modifying the sync code to do its thing one directory at a time, therefore enormously reducing its memory use for large dirtrees.

Nick, is there any chance of you doing that same modification for rclone cryptcheck? Should I open a github issue for it?

Thanks,
– Durval,

2

Yes, please make a new issue on github :slight_smile:

rclone check and rclone cryptcheck should use the same machinery as rclone sync/copy/move for traversing two directory trees in step. I haven’t managed to generalise it yet though.

3

I had a go at the refactor - it turned out quite well I think.

Would you mind having a go with a beta before I merge it to mainline? All the unit tests pass so it should be good!

https://pub.rclone.org/v1.37-120-g08217c79-sync-refactorβ/

4

Hi Nick,

Done: https://github.com/ncw/rclone/issues/1657

Cheers,
– Done.

5

Wow, @ncw! You did it before I could come back and see your first reply (re: opening the github issue)!! Thank you very much for the great service!!! And in a related note, if they ever start canonizing Open Source Saints, I will immediately propose your name :wink:

Seriously, this is way better than what I’m used to see from closed-source companies who have a lot of resources and charge a lot of money not only for their products but also additionally for supporting them (cough Oracle cough). Seriously again, if you ever decide to open a Patreon account so I can contribute to funding these flowers for your wife on a regular basis (or whatever else you think is worthy), just let me know.

Sure thing! The disks involved are currently offline (I had to repurpose the machine they were on for another task), but I will reassemble the whole she-bang and get the beta running there.

Please give me a few days, not only for reassembling that setup, but also because this dirtree is huge and will take at least a few days to be processed (I guess, never used rsync cryptcheck on anything that large before).

1 Like