Memory leak with Mega?

cristianlivella · February 14, 2021, 11:24pm

What is the problem you are having with rclone?

I started using rclone some days ago to backup an hard disk on Mega. I'm using a Raspberry PI with 1 GB ram, everything was going pretty well, but I had to stop the process due some issue with Mega back end (I guess API rate limit), and now when I tried to run rclone again, all the Raspberry ram is filled in a few minutes, even before the files transfer actually starts, and rclone fail.
At the moment I have a lot of file on Mega, could it be that the problem is caused by the list of files that does not fit in the ram? Is there any solution to this?

What is your rclone version (output from `rclone version`)

rclone v1.54.0

os/arch: linux/arm
go version: go1.15.7

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Raspbian 9 - 32 bit

Which cloud storage system are you using? (eg Google Drive)

Mega with Crypt

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

rclone sync -i /mnt/disk crypt:/disk --transfers=1 --checkers=1 --buffer-size=0 --max-backlog=1 --tpslimit=1 --cache-chunk-no-memory

A log from the command

pi@raspberry:~ $ rclone sync -i /mnt/disk crypt:/disk --transfers=1 --checkers=1 --buffer-size=0 --max-backlog=1 --tpslimit=1 --cache-chunk-no-memory
runtime: out of memory: cannot allocate 4194304-byte block (1803124736 in use)
fatal error: out of memory
  
goroutine 1 [running]:
runtime.throw(0x13ac5e8, 0xd) 
runtime/panic.go:1116 +0x5c fp=0x248f5e0 sp=0x248f5cc pc=0x47024  
runtime.(*mcache).refill(0x76f09088, 0xb) 
runtime/mcache.go:144 +0x100 fp=0x248f5f4 sp=0x248f5e0 pc=0x265b8 
runtime.(*mcache).nextFree(0x76f09088, 0x226540b, 0x1637585, 0x6d93faf2, 0x28)
runtime/malloc.go:880 +0x7c fp=0x248f614 sp=0x248f5f4 pc=0x1be20  
runtime.mallocgc(0x40, 0x113e970, 0x1, 0x2c)  
runtime/malloc.go:1061 +0x7c8 fp=0x248f67c sp=0x248f614 pc=0x1c794
runtime.makeslice(0x113e970, 0x10, 0x40, 0xa3f09c)
runtime/slice.go:98 +0x6c fp=0x248f690 sp=0x248f67c pc=0x5f7d4
bytes.(*Buffer).grow(0x7e7fa7e0, 0x10, 0xa40700)  
bytes/buffer.go:128 +0x1fc fp=0x248f6b4 sp=0x248f690 pc=0x15850c  
bytes.(*Buffer).Grow(...) 
bytes/buffer.go:161   
github.com/t3rm1n4l/go-mega.a32_to_bytes(0x248f714, 0x4, 0x4, 0x166d5c0, 0x7e7fa7c0, 0x0, 0x0, 0x10)  
github.com/t3rm1n4l/go-mega@v0.0.0-20200416171014-ffad7fcb44b8/utils.go:58 +0x40 fp=0x248f6e0 sp=0x248f6b4 pc=0xa3f0c0
github.com/t3rm1n4l/go-mega.decryptAttr(0x7e7f3fc0, 0x10, 0x40, 0x5078780, 0x80, 0x40, 0x0, 0x0, 0x0) 
github.com/t3rm1n4l/go-mega@v0.0.0-20200416171014-ffad7fcb44b8/utils.go:316 +0x84 fp=0x248f73c sp=0x248f6e0 pc=0xa40744   
github.com/t3rm1n4l/go-mega.(*Mega).addFSNode(0x25082d0, 0x5076c50, 0x8, 0x5076c58, 0x8, 0x5076c60, 0xb, 0x0, 0x5078780, 0x80, ...)   
github.com/t3rm1n4l/go-mega@v0.0.0-20200416171014-ffad7fcb44b8/mega.go:808 +0xd94 fp=0x248f920 sp=0x248f73c pc=0xa37a84   
github.com/t3rm1n4l/go-mega.(*Mega).getFileSystem(0x25082d0, 0x0, 0x0)
github.com/t3rm1n4l/go-mega@v0.0.0-20200416171014-ffad7fcb44b8/mega.go:934 +0x250 fp=0x248faa4 sp=0x248f920 pc=0xa38330   
github.com/t3rm1n4l/go-mega.(*Mega).Login(0x25082d0, 0x29b4047, 0x18, 0x27a7bc0, 0x3c, 0x0, 0x4)  
github.com/t3rm1n4l/go-mega@v0.0.0-20200416171014-ffad7fcb44b8/mega.go:595 +0x98 fp=0x248fac8 sp=0x248faa4 pc=0xa36568
github.com/rclone/rclone/backend/mega.NewFs(0x1675e20, 0x250c000, 0x276e780, 0x4, 0x276e785, 0x25, 0x1668240, 0x28fe240, 0x0, 0x0, ...)
github.com/rclone/rclone/backend/mega/mega.go:219 +0x59c fp=0x248fb3c sp=0x248fac8 pc=0xa44f90
github.com/rclone/rclone/fs.NewFs(0x1675e20, 0x250c000, 0x276e780, 0x2a, 0x24a0b60, 0x1, 0x0, 0x87c01)
github.com/rclone/rclone/fs/fs.go:1366 +0x10c fp=0x248fb78 sp=0x248fb3c pc=0x338584
github.com/rclone/rclone/fs/cache.GetFn.func1(0x276e780, 0x2a, 0x276e780, 0x2a, 0x1ff9c58, 0x0, 0x0)
github.com/rclone/rclone/fs/cache/cache.go:48 +0x44 fp=0x248fbb0 sp=0x248fb78 pc=0x3419b0
github.com/rclone/rclone/lib/cache.(*Cache).Get(0x24a0b60, 0x276e780, 0x2a, 0x27dbc38, 0x0, 0x0, 0x0, 0x0)
github.com/rclone/rclone/lib/cache/cache.go:61 +0x114 fp=0x248fbe0 sp=0x248fbb0 pc=0x340530
github.com/rclone/rclone/fs/cache.GetFn(0x1675e20, 0x250c000, 0x276e780, 0x2a, 0x146de10, 0x2a, 0xa, 0x0, 0x8)
github.com/rclone/rclone/fs/cache/cache.go:47 +0x98 fp=0x248fc4c sp=0x248fbe0 pc=0x3413d4
github.com/rclone/rclone/fs/cache.Get(...)
github.com/rclone/rclone/fs/cache/cache.go:104
github.com/rclone/rclone/backend/crypt.NewFs(0x1675e20, 0x250c000, 0x7e9967c5, 0x9, 0x7e9967cf, 0x8, 0x1668240, 0x28fe640, 0x0, 0x0, ...)
github.com/rclone/rclone/backend/crypt/crypt.go:172 +0x200 fp=0x248fcfc sp=0x248fc4c pc=0x6055a0
github.com/rclone/rclone/fs.NewFs(0x1675e20, 0x250c000, 0x7e9967c5, 0x12, 0x24a0b60, 0x1, 0x0, 0x87c01)
github.com/rclone/rclone/fs/fs.go:1366 +0x10c fp=0x248fd38 sp=0x248fcfc pc=0x338584
github.com/rclone/rclone/fs/cache.GetFn.func1(0x7e9967c5, 0x12, 0x7e9967c5, 0x12, 0x1ff9c58, 0x0, 0x0)
github.com/rclone/rclone/fs/cache/cache.go:48 +0x44 fp=0x248fd70 sp=0x248fd38 pc=0x3419b0
github.com/rclone/rclone/lib/cache.(*Cache).Get(0x24a0b60, 0x7e9967c5, 0x12, 0x2961df8, 0x0, 0x0, 0x0, 0x0)
github.com/rclone/rclone/lib/cache/cache.go:61 +0x114 fp=0x248fda0 sp=0x248fd70 pc=0x340530
github.com/rclone/rclone/fs/cache.GetFn(0x1675e20, 0x250c000, 0x7e9967c5, 0x12, 0x146de10, 0x10, 0x1688e50, 0x2939b00, 0xea8d18)
github.com/rclone/rclone/fs/cache/cache.go:47 +0x98 fp=0x248fe0c sp=0x248fda0 pc=0x3413d4
github.com/rclone/rclone/fs/cache.Get(...)
github.com/rclone/rclone/fs/cache/cache.go:104
github.com/rclone/rclone/cmd.newFsDir(0x7e9967c5, 0x12, 0x1688e50, 0x2688310)
github.com/rclone/rclone/cmd/cmd.go:142 +0x40 fp=0x248fe54 sp=0x248fe0c pc=0xef9100
github.com/rclone/rclone/cmd.NewFsSrcFileDst(0x260e900, 0x2, 0xb, 0x260e900, 0x2, 0xb, 0x261fa80, 0x139d392, 0x113e790)
github.com/rclone/rclone/cmd/cmd.go:171 +0x78 fp=0x248fe80 sp=0x248fe54 pc=0xef9338
github.com/rclone/rclone/cmd/sync.glob..func1(0x1fdd630, 0x260e900, 0x2, 0xb)
github.com/rclone/rclone/cmd/sync/sync.go:52 +0x5c fp=0x248fec4 sp=0x248fe80 pc=0x1016448
github.com/spf13/cobra.(*Command).execute(0x1fdd630, 0x260e780, 0xb, 0xc, 0x1fdd630, 0x260e780)
github.com/spf13/cobra@v1.1.1/command.go:854 +0x1e8 fp=0x248ff2c sp=0x248fec4 pc=0xeedf74
github.com/spf13/cobra.(*Command).ExecuteC(0x1fd7a50, 0x0, 0x2500030, 0x0)
github.com/spf13/cobra@v1.1.1/command.go:958 +0x274 fp=0x248ff94 sp=0x248ff2c pc=0xeee90c
github.com/spf13/cobra.(*Command).Execute(...)
github.com/spf13/cobra@v1.1.1/command.go:895
github.com/rclone/rclone/cmd.Main()
github.com/rclone/rclone/cmd/cmd.go:543 +0x90 fp=0x248ffb4 sp=0x248ff94 pc=0xefbb40
main.main()
github.com/rclone/rclone/rclone.go:14 +0x14 fp=0x248ffb8 sp=0x248ffb4 pc=0x101da44
runtime.main()
runtime/proc.go:204 +0x208 fp=0x248ffe4 sp=0x248ffb8 pc=0x49e78
runtime.goexit()
runtime/asm_arm.s:857 +0x4 fp=0x248ffe4 sp=0x248ffe4 pc=0x7f1dc

goroutine 22 [select]:
go.opencensus.io/stats/view.(*worker).start(0x25ccb40)
go.opencensus.io@v0.22.5/stats/view/worker.go:276 +0xac
created by go.opencensus.io/stats/view.init.0
go.opencensus.io@v0.22.5/stats/view/worker.go:34 +0x58

asdffdsa · February 14, 2021, 11:42pm

hello and welcome to the forum,

which pi hardware do you have?
32bit can only access 4GB

this might help
https://rclone.org/docs/#use-mmap

no config file was posted, but i assume/hope that you are not using https://rclone.org/cache/
if true, then this flag does nothing
--cache-chunk-no-memory

cristianlivella · February 14, 2021, 11:46pm

Hi! I'm using a Raspberry Pi 3B+ with 1 GB ram.

I didn't posted config file because it's encrypted, anyway I didn't do any special configuration, so I guess cache is not enabled.

I'm now trying with use-mmap, thanks

edit: I cannot see any improvement, the process stops in less then a minute.

asdffdsa · February 14, 2021, 11:52pm

no worries, sooner than later, one of our rclone experts will stop by....

ncw · February 15, 2021, 5:42pm

That could be the problem. How many files do you have do you think?

The mega backend has to load all the file objects into RAM which is quite memory intensive.

Another solution would be to use megacmd to serve a webdav backend and connect rclone to that. There is an issue with some help on how to do this here: https://github.com/rclone/rclone/issues/3660

asdffdsa · February 15, 2021, 6:15pm

hi,

would this beta be helpful for testing in this case?
Guesstimating memory consumption of --fast-list - #2 by ncw

cristianlivella · February 15, 2021, 8:45pm

Thanks @ncw, I'm trying with megacmd webdav and it seems work better, the ram usage stays pretty high, but at least I can use it.

My main concern now is that it seems megacmd makes a copy of the files in a local directory, before uploading it, and I may have some file on the external drive that are bigger than the Raspberry SD card.

Do you know if megacmd handle properly situation like that? Or is better if I use the rclone chunker?

edit: does rclone chunker make chunks one by one, or all the chunks are created before the upload? Because in this case I guess I'll have the same issue of storage...

ncw · February 16, 2021, 4:40pm

I don't know. I think it buffers the files on disk for any size.

Chunker will upload a few chunks in parallel but not many so this shouldn't be a problem.

cristianlivella · February 16, 2021, 5:44pm

That looks good. Thanks again, I wish I had discovered rclone before, it seems a well designed and working project and I see you are putting a lot of effort in it!

system · February 19, 2021, 5:44pm

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.