Memory leak with Mega?

What is the problem you are having with rclone?

I started using rclone some days ago to backup an hard disk on Mega. I'm using a Raspberry PI with 1 GB ram, everything was going pretty well, but I had to stop the process due some issue with Mega back end (I guess API rate limit), and now when I tried to run rclone again, all the Raspberry ram is filled in a few minutes, even before the files transfer actually starts, and rclone fail.
At the moment I have a lot of file on Mega, could it be that the problem is caused by the list of files that does not fit in the ram? Is there any solution to this?

What is your rclone version (output from rclone version)

rclone v1.54.0

  • os/arch: linux/arm
  • go version: go1.15.7

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Raspbian 9 - 32 bit

Which cloud storage system are you using? (eg Google Drive)

Mega with Crypt

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone sync -i /mnt/disk crypt:/disk --transfers=1 --checkers=1 --buffer-size=0 --max-backlog=1 --tpslimit=1 --cache-chunk-no-memory

A log from the command

pi@raspberry:~ $ rclone sync -i /mnt/disk crypt:/disk --transfers=1 --checkers=1 --buffer-size=0 --max-backlog=1 --tpslimit=1 --cache-chunk-no-memory
runtime: out of memory: cannot allocate 4194304-byte block (1803124736 in use)
fatal error: out of memory
goroutine 1 [running]:
runtime.throw(0x13ac5e8, 0xd) 
runtime/panic.go:1116 +0x5c fp=0x248f5e0 sp=0x248f5cc pc=0x47024  
runtime.(*mcache).refill(0x76f09088, 0xb) 
runtime/mcache.go:144 +0x100 fp=0x248f5f4 sp=0x248f5e0 pc=0x265b8 
runtime.(*mcache).nextFree(0x76f09088, 0x226540b, 0x1637585, 0x6d93faf2, 0x28)
runtime/malloc.go:880 +0x7c fp=0x248f614 sp=0x248f5f4 pc=0x1be20  
runtime.mallocgc(0x40, 0x113e970, 0x1, 0x2c)  
runtime/malloc.go:1061 +0x7c8 fp=0x248f67c sp=0x248f614 pc=0x1c794
runtime.makeslice(0x113e970, 0x10, 0x40, 0xa3f09c)
runtime/slice.go:98 +0x6c fp=0x248f690 sp=0x248f67c pc=0x5f7d4
bytes.(*Buffer).grow(0x7e7fa7e0, 0x10, 0xa40700)  
bytes/buffer.go:128 +0x1fc fp=0x248f6b4 sp=0x248f690 pc=0x15850c  
bytes/buffer.go:161, 0x4, 0x4, 0x166d5c0, 0x7e7fa7c0, 0x0, 0x0, 0x10) +0x40 fp=0x248f6e0 sp=0x248f6b4 pc=0xa3f0c0, 0x10, 0x40, 0x5078780, 0x80, 0x40, 0x0, 0x0, 0x0) +0x84 fp=0x248f73c sp=0x248f6e0 pc=0xa40744*Mega).addFSNode(0x25082d0, 0x5076c50, 0x8, 0x5076c58, 0x8, 0x5076c60, 0xb, 0x0, 0x5078780, 0x80, ...) +0xd94 fp=0x248f920 sp=0x248f73c pc=0xa37a84*Mega).getFileSystem(0x25082d0, 0x0, 0x0) +0x250 fp=0x248faa4 sp=0x248f920 pc=0xa38330*Mega).Login(0x25082d0, 0x29b4047, 0x18, 0x27a7bc0, 0x3c, 0x0, 0x4) +0x98 fp=0x248fac8 sp=0x248faa4 pc=0xa36568, 0x250c000, 0x276e780, 0x4, 0x276e785, 0x25, 0x1668240, 0x28fe240, 0x0, 0x0, ...) +0x59c fp=0x248fb3c sp=0x248fac8 pc=0xa44f90, 0x250c000, 0x276e780, 0x2a, 0x24a0b60, 0x1, 0x0, 0x87c01) +0x10c fp=0x248fb78 sp=0x248fb3c pc=0x338584, 0x2a, 0x276e780, 0x2a, 0x1ff9c58, 0x0, 0x0) +0x44 fp=0x248fbb0 sp=0x248fb78 pc=0x3419b0*Cache).Get(0x24a0b60, 0x276e780, 0x2a, 0x27dbc38, 0x0, 0x0, 0x0, 0x0) +0x114 fp=0x248fbe0 sp=0x248fbb0 pc=0x340530, 0x250c000, 0x276e780, 0x2a, 0x146de10, 0x2a, 0xa, 0x0, 0x8) +0x98 fp=0x248fc4c sp=0x248fbe0 pc=0x3413d4, 0x250c000, 0x7e9967c5, 0x9, 0x7e9967cf, 0x8, 0x1668240, 0x28fe640, 0x0, 0x0, ...) +0x200 fp=0x248fcfc sp=0x248fc4c pc=0x6055a0, 0x250c000, 0x7e9967c5, 0x12, 0x24a0b60, 0x1, 0x0, 0x87c01) +0x10c fp=0x248fd38 sp=0x248fcfc pc=0x338584, 0x12, 0x7e9967c5, 0x12, 0x1ff9c58, 0x0, 0x0) +0x44 fp=0x248fd70 sp=0x248fd38 pc=0x3419b0*Cache).Get(0x24a0b60, 0x7e9967c5, 0x12, 0x2961df8, 0x0, 0x0, 0x0, 0x0) +0x114 fp=0x248fda0 sp=0x248fd70 pc=0x340530, 0x250c000, 0x7e9967c5, 0x12, 0x146de10, 0x10, 0x1688e50, 0x2939b00, 0xea8d18) +0x98 fp=0x248fe0c sp=0x248fda0 pc=0x3413d4, 0x12, 0x1688e50, 0x2688310) +0x40 fp=0x248fe54 sp=0x248fe0c pc=0xef9100, 0x2, 0xb, 0x260e900, 0x2, 0xb, 0x261fa80, 0x139d392, 0x113e790) +0x78 fp=0x248fe80 sp=0x248fe54 pc=0xef9338, 0x260e900, 0x2, 0xb) +0x5c fp=0x248fec4 sp=0x248fe80 pc=0x1016448*Command).execute(0x1fdd630, 0x260e780, 0xb, 0xc, 0x1fdd630, 0x260e780) +0x1e8 fp=0x248ff2c sp=0x248fec4 pc=0xeedf74*Command).ExecuteC(0x1fd7a50, 0x0, 0x2500030, 0x0) +0x274 fp=0x248ff94 sp=0x248ff2c pc=0xeee90c*Command).Execute(...) +0x90 fp=0x248ffb4 sp=0x248ff94 pc=0xefbb40
main.main() +0x14 fp=0x248ffb8 sp=0x248ffb4 pc=0x101da44
runtime/proc.go:204 +0x208 fp=0x248ffe4 sp=0x248ffb8 pc=0x49e78
runtime/asm_arm.s:857 +0x4 fp=0x248ffe4 sp=0x248ffe4 pc=0x7f1dc

goroutine 22 [select]:*worker).start(0x25ccb40) +0xac
created by +0x58

hello and welcome to the forum,

which pi hardware do you have?
32bit can only access 4GB

this might help

no config file was posted, but i assume/hope that you are not using
if true, then this flag does nothing

Hi! I'm using a Raspberry Pi 3B+ with 1 GB ram.

I didn't posted config file because it's encrypted, anyway I didn't do any special configuration, so I guess cache is not enabled.

I'm now trying with use-mmap, thanks :slight_smile:

edit: I cannot see any improvement, the process stops in less then a minute.

no worries, sooner than later, one of our rclone experts will stop by....

That could be the problem. How many files do you have do you think?

The mega backend has to load all the file objects into RAM which is quite memory intensive.

Another solution would be to use megacmd to serve a webdav backend and connect rclone to that. There is an issue with some help on how to do this here:

1 Like


would this beta be helpful for testing in this case?
Guesstimating memory consumption of --fast-list - #2 by ncw

Thanks @ncw, I'm trying with megacmd webdav and it seems work better, the ram usage stays pretty high, but at least I can use it.

My main concern now is that it seems megacmd makes a copy of the files in a local directory, before uploading it, and I may have some file on the external drive that are bigger than the Raspberry SD card.

Do you know if megacmd handle properly situation like that? Or is better if I use the rclone chunker?

edit: does rclone chunker make chunks one by one, or all the chunks are created before the upload? Because in this case I guess I'll have the same issue of storage...

I don't know. I think it buffers the files on disk for any size.

Chunker will upload a few chunks in parallel but not many so this shouldn't be a problem.

That looks good. Thanks again, I wish I had discovered rclone before, it seems a well designed and working project and I see you are putting a lot of effort in it!

1 Like

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.