How to set a memory limit?

Help and Support
1

hello,

I run command:
rclone copy --ignore-existing -P --exclude-from "exclude.txt" "/home/my" MEGA:my

How to set a memory limit?

Transferred:      484.004k / 4.054 MBytes, 12%, 189 Bytes/s, ETA 5h30m51s
Errors:                 0
Checks:            151251 / 151251, 100%
Transferred:           37 / 10042, 0%
Elapsed time:      43m40s
Transferring:
 *             my/sitemap/website/9732.html:100% /1, 0/s, -
 * my/image/compress/…ture/my/2839865.jpg:100% /14.227k, 1.387k/s, 0s
 * my/image/compress/…ture/my/2839866.jpg:100% /11.308k, 1.312k/s, 0s
 * my/image/compress/…ture/my/2839867.jpg:100% /8.544k, 8.146k/s, 0s

Next errors:

fatal error: runtime: out of memory

runtime stack:
runtime.throw(0x138ddca, 0x16)
        /home/travis/.gimme/versions/go1.12.4.linux.amd64/src/runtime/panic.go:617 +0x72
runtime.sysMap(0xc054000000, 0x4000000, 0x20a5798)
        /home/travis/.gimme/versions/go1.12.4.linux.amd64/src/runtime/mem_linux.go:170 +0xc7
runtime.(*mheap).sysAlloc(0x208cb80, 0x6f0000, 0x208cb90, 0x378)
        /home/travis/.gimme/versions/go1.12.4.linux.amd64/src/runtime/malloc.go:633 +0x1cd
runtime.(*mheap).grow(0x208cb80, 0x378, 0x0)
        /home/travis/.gimme/versions/go1.12.4.linux.amd64/src/runtime/mheap.go:1232 +0x42
runtime.(*mheap).allocSpanLocked(0x208cb80, 0x378, 0x20a57a8, 0x41d5ed)
        /home/travis/.gimme/versions/go1.12.4.linux.amd64/src/runtime/mheap.go:1150 +0x3a7
runtime.(*mheap).alloc_m(0x208cb80, 0x378, 0x7f5319430100, 0x208cb90)
        /home/travis/.gimme/versions/go1.12.4.linux.amd64/src/runtime/mheap.go:977 +0xc2
runtime.(*mheap).alloc.func1()
        /home/travis/.gimme/versions/go1.12.4.linux.amd64/src/runtime/mheap.go:1048 +0x4c
runtime.(*mheap).alloc(0x208cb80, 0x378, 0xc000010100, 0x7fffb9ba5c9b)
        /home/travis/.gimme/versions/go1.12.4.linux.amd64/src/runtime/mheap.go:1047 +0x8a
runtime.largeAlloc(0x6f0000, 0x1, 0xc00372efb8)
        /home/travis/.gimme/versions/go1.12.4.linux.amd64/src/runtime/malloc.go:1055 +0x99
runtime.mallocgc.func1()
        /home/travis/.gimme/versions/go1.12.4.linux.amd64/src/runtime/malloc.go:950 +0x46
runtime.systemstack(0xc0000a26c0)
        /home/travis/.gimme/versions/go1.12.4.linux.amd64/src/runtime/asm_amd64.s:351 +0x66
runtime.mstart()
        /home/travis/.gimme/versions/go1.12.4.linux.amd64/src/runtime/proc.go:1153

goroutine 68 [running]:
runtime.systemstack_switch()
        /home/travis/.gimme/versions/go1.12.4.linux.amd64/src/runtime/asm_amd64.s:311 fp=0xc$
runtime.mallocgc(0x6f0000, 0x1218820, 0x1, 0x2)
        /home/travis/.gimme/versions/go1.12.4.linux.amd64/src/runtime/malloc.go:949 +0x872 f$
runtime.growslice(0x1218820, 0xc053a22000, 0x2c600, 0x2c600, 0x2c601, 0xc00372fba8, 0x1, 0x1)
        /home/travis/.gimme/versions/go1.12.4.linux.amd64/src/runtime/slice.go:181 +0x1e6 fp$
github.com/ncw/rclone/fs/march.matchListings(0xc048af4000, 0x32f77, 0x38c00, 0xc038000000, 0$
        /home/travis/gopath/src/github.com/ncw/rclone/fs/march/march.go:324 +0x3ad fp=0xc003$
github.com/ncw/rclone/fs/march.(*March).processJob(0xc0000a6b00, 0xc009cee0c0, 0x13, 0xc0003$
        /home/travis/gopath/src/github.com/ncw/rclone/fs/march/march.go:388 +0x22e fp=0xc003$
github.com/ncw/rclone/fs/march.(*March).Run.func1(0xc004cc3ab0, 0xc0000a6b00, 0xc0002f6480, $
        /home/travis/gopath/src/github.com/ncw/rclone/fs/march/march.go:141 +0x1f1 fp=0xc003$
runtime.goexit()
        /home/travis/.gimme/versions/go1.12.4.linux.amd64/src/runtime/asm_amd64.s:1337 +0x1 $
created by github.com/ncw/rclone/fs/march.(*March).Run
        /home/travis/gopath/src/github.com/ncw/rclone/fs/march/march.go:131 +0x14a

goroutine 1 [semacquire]:
sync.runtime_Semacquire(0xc004cc3b48)
        /home/travis/.gimme/versions/go1.12.4.linux.amd64/src/runtime/sema.go:56 +0x39
sync.(*WaitGroup).Wait(0xc004cc3b40)
        /home/travis/.gimme/versions/go1.12.4.linux.amd64/src/sync/waitgroup.go:130 +0x65
github.com/ncw/rclone/fs/march.(*March).Run(0xc0000a6b00)
        /home/travis/gopath/src/github.com/ncw/rclone/fs/march/march.go:178 +0x25c
github.com/ncw/rclone/fs/sync.(*syncCopyMove).run(0xc0004356c0, 0xc0000a6980, 0x1614200)
        /home/travis/gopath/src/github.com/ncw/rclone/fs/sync/sync.go:673 +0x204
github.com/ncw/rclone/fs/sync.runSyncCopyMove(0x1614300, 0xc0000a6980, 0x1614200, 0xc0000315$
        /home/travis/gopath/src/github.com/ncw/rclone/fs/sync/sync.go:884 +0x236
github.com/ncw/rclone/fs/sync.CopyDir(...)
        /home/travis/gopath/src/github.com/ncw/rclone/fs/sync/sync.go:894
github.com/ncw/rclone/cmd/copy.glob..func1.1(0xc0005a60e0, 0x15dec40)
        /home/travis/gopath/src/github.com/ncw/rclone/cmd/copy/copy.go:77 +0x6e
github.com/ncw/rclone/cmd.Run(0xc0002f0101, 0x20737c0, 0xc011a3dd40)
        /home/travis/gopath/src/github.com/ncw/rclone/cmd/cmd.go:232 +0xc7
github.com/ncw/rclone/cmd/copy.glob..func1(0x20737c0, 0xc0002f6d80, 0x2, 0x6)
        /home/travis/gopath/src/github.com/ncw/rclone/cmd/copy/copy.go:75 +0x129
github.com/ncw/rclone/vendor/github.com/spf13/cobra.(*Command).execute(0x20737c0, 0xc0002f6c$
        /home/travis/gopath/src/github.com/ncw/rclone/vendor/github.com/spf13/cobra/command.$
github.com/ncw/rclone/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0x2070840, 0xc0002e9$
        /home/travis/gopath/src/github.com/ncw/rclone/vendor/github.com/spf13/cobra/command.$
github.com/ncw/rclone/vendor/github.com/spf13/cobra.(*Command).Execute(...)
        /home/travis/gopath/src/github.com/ncw/rclone/vendor/github.com/spf13/cobra/command.$
github.com/ncw/rclone/cmd.Main()
        /home/travis/gopath/src/github.com/ncw/rclone/cmd/cmd.go:488 +0x47
main.main()
        /home/travis/gopath/src/github.com/ncw/rclone/rclone.go:13 +0x20

goroutine 18 [syscall, 43 minutes]:
os/signal.signal_recv(0x0)
        /home/travis/.gimme/versions/go1.12.4.linux.amd64/src/runtime/sigqueue.go:139 +0x9c
os/signal.loop()
        /home/travis/.gimme/versions/go1.12.4.linux.amd64/src/os/signal/signal_unix.go:23 +0$
created by os/signal.init.0
        /home/travis/.gimme/versions/go1.12.4.linux.amd64/src/os/signal/signal_unix.go:29 +0$

goroutine 54 [select]:
github.com/ncw/rclone/fs/sync.(*pipe).Get(0xc00065c700, 0x1601b40, 0xc00065c7c0, 0xc0339535c$
        /home/travis/gopath/src/github.com/ncw/rclone/fs/sync/pipe.go:62 +0xe4
github.com/ncw/rclone/fs/sync.(*syncCopyMove).pairChecker(0xc0004356c0, 0xc00065c700, 0xc000$
        /home/travis/gopath/src/github.com/ncw/rclone/fs/sync/sync.go:213 +0xb0
created by github.com/ncw/rclone/fs/sync.(*syncCopyMove).startCheckers
        /home/travis/gopath/src/github.com/ncw/rclone/fs/sync/sync.go:306 +0x9a

goroutine 52 [select]:
github.com/ncw/rclone/fs/sync.(*pipe).Get(0xc00065c700, 0x1601b40, 0xc00065c7c0, 0xc00eb9bb0$
        /home/travis/gopath/src/github.com/ncw/rclone/fs/sync/pipe.go:62 +0xe4
github.com/ncw/rclone/fs/sync.(*syncCopyMove).pairChecker(0xc0004356c0, 0xc00065c700, 0xc000$
        /home/travis/gopath/src/github.com/ncw/rclone/fs/sync/sync.go:213 +0xb0
created by github.com/ncw/rclone/fs/sync.(*syncCopyMove).startCheckers
        /home/travis/gopath/src/github.com/ncw/rclone/fs/sync/sync.go:306 +0x9a

goroutine 51 [select]:
github.com/ncw/rclone/fs/sync.(*pipe).Get(0xc00065c700, 0x1601b40, 0xc00065c7c0, 0xc0107a992$
        /home/travis/gopath/src/github.com/ncw/rclone/fs/sync/pipe.go:62 +0xe4
github.com/ncw/rclone/fs/sync.(*syncCopyMove).pairChecker(0xc0004356c0, 0xc00065c700, 0xc000$
        /home/travis/gopath/src/github.com/ncw/rclone/fs/sync/sync.go:213 +0xb0
created by github.com/ncw/rclone/fs/sync.(*syncCopyMove).startCheckers
        /home/travis/gopath/src/github.com/ncw/rclone/fs/sync/sync.go:306 +0x9a

goroutine 53 [select]:
github.com/ncw/rclone/fs/sync.(*pipe).Get(0xc00065c700, 0x1601b40, 0xc00065c7c0, 0xc033a329c$
        /home/travis/gopath/src/github.com/ncw/rclone/fs/sync/pipe.go:62 +0xe4
github.com/ncw/rclone/fs/sync.(*syncCopyMove).pairChecker(0xc0004356c0, 0xc00065c700, 0xc000$
        /home/travis/gopath/src/github.com/ncw/rclone/fs/sync/sync.go:213 +0xb0
created by github.com/ncw/rclone/fs/sync.(*syncCopyMove).startCheckers
        /home/travis/gopath/src/github.com/ncw/rclone/fs/sync/sync.go:306 +0x9a

goroutine 27 [select]:
net/http.(*persistConn).roundTrip(0xc0001f50e0, 0xc00554ebd0, 0x0, 0x0, 0x0)
        /home/travis/.gimme/versions/go1.12.4.linux.amd64/src/net/http/transport.go:2174 +0x$
net/http.(*Transport).roundTrip(0xc0001f5e60, 0xc0016b9900, 0x137c7c5, 0xa, 0xc0017eae48)
        /home/travis/.gimme/versions/go1.12.4.linux.amd64/src/net/http/transport.go:481 +0xa$
net/http.(*Transport).RoundTrip(...)
        /home/travis/.gimme/versions/go1.12.4.linux.amd64/src/net/http/roundtrip.go:17
github.com/ncw/rclone/fs/fshttp.(*Transport).RoundTrip(0xc0003c4bd0, 0xc0016b9900, 0xc0003c4$
        /home/travis/gopath/src/github.com/ncw/rclone/fs/fshttp/http.go:324 +0x1e3
net/http.send(0xc0016b9900, 0x15dec40, 0xc0003c4bd0, 0x0, 0x0, 0x0, 0xc0000a16f8, 0xc018a1fc$
        /home/travis/.gimme/versions/go1.12.4.linux.amd64/src/net/http/client.go:250 +0x461
net/http.(*Client).send(0xc0003c4c00, 0xc0016b9900, 0x0, 0x0, 0x0, 0xc0000a16f8, 0x0, 0x1, 0$
        /home/travis/.gimme/versions/go1.12.4.linux.amd64/src/net/http/client.go:174 +0xfb
net/http.(*Client).do(0xc0003c4c00, 0xc0016b9900, 0x0, 0x0, 0x0)
        /home/travis/.gimme/versions/go1.12.4.linux.amd64/src/net/http/client.go:641 +0x279
net/http.(*Client).Do(...)
        /home/travis/.gimme/versions/go1.12.4.linux.amd64/src/net/http/client.go:509
net/http.(*Client).Get(0xc0003c4c00, 0xc008d70c80, 0x33, 0x10ec0e0, 0xc0017d6440, 0x0)
        /home/travis/.gimme/versions/go1.12.4.linux.amd64/src/net/http/client.go:398 +0x9e
github.com/ncw/rclone/vendor/github.com/t3rm1n4l/go-mega.(*Mega).pollEvents(0xc00033fa00)
        /home/travis/gopath/src/github.com/ncw/rclone/vendor/github.com/t3rm1n4l/go-mega/meg$
created by github.com/ncw/rclone/vendor/github.com/t3rm1n4l/go-mega.(*Mega).getFileSystem
        /home/travis/gopath/src/github.com/ncw/rclone/vendor/github.com/t3rm1n4l/go-mega/meg$

goroutine 49 [select]:
github.com/ncw/rclone/cmd.startProgress.func3(0xc004cc3a50, 0xc0000808a0, 0x13f0170)
        /home/travis/gopath/src/github.com/ncw/rclone/cmd/progress.go:54 +0x116
created by github.com/ncw/rclone/cmd.startProgress
        /home/travis/gopath/src/github.com/ncw/rclone/cmd/progress.go:46 +0xd1

goroutine 15 [IO wait]:
internal/poll.runtime_pollWait(0x7f531a6dbdd8, 0x72, 0xffffffffffffffff)
        /home/travis/.gimme/versions/go1.12.4.linux.amd64/src/runtime/netpoll.go:182 +0x56
internal/poll.(*pollDesc).wait(0xc0000a6718, 0x72, 0x1500, 0x15c9, 0xffffffffffffffff)
        /home/travis/.gimme/versions/go1.12.4.linux.amd64/src/internal/poll/fd_poll_runtime.$
internal/poll.(*pollDesc).waitRead(...)
        /home/travis/.gimme/versions/go1.12.4.linux.amd64/src/internal/poll/fd_poll_runtime.$
internal/poll.(*FD).Read(0xc0000a6700, 0xc000339000, 0x15c9, 0x15c9, 0x0, 0x0, 0x0)
        /home/travis/.gimme/versions/go1.12.4.linux.amd64/src/internal/poll/fd_unix.go:169 +$
net.(*netFD).Read(0xc0000a6700, 0xc000339000, 0x15c9, 0x15c9, 0x6dcb238b84b313ec, 0x6258e8a9$
        /home/travis/.gimme/versions/go1.12.4.linux.amd64/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc00000e178, 0xc000339000, 0x15c9, 0x15c9, 0x0, 0x0, 0x0)
        /home/travis/.gimme/versions/go1.12.4.linux.amd64/src/net/net.go:177 +0x69
github.com/ncw/rclone/fs/fshttp.(*timeoutConn).readOrWrite(0xc015be2dc0, 0xc022f46850, 0xc00$
        /home/travis/gopath/src/github.com/ncw/rclone/fs/fshttp/http.go:75 +0x48
github.com/ncw/rclone/fs/fshttp.(*timeoutConn).Read(0xc015be2dc0, 0xc000339000, 0x15c9, 0x15$
        /home/travis/gopath/src/github.com/ncw/rclone/fs/fshttp/http.go:87 +0x8a
crypto/tls.(*atLeastReader).Read(0xc04cb95aa0, 0xc000339000, 0x15c9, 0x15c9, 0x1, 0xa, 0xc02$
        /home/travis/.gimme/versions/go1.12.4.linux.amd64/src/crypto/tls/conn.go:761 +0x60
bytes.(*Buffer).ReadFrom(0xc000059758, 0x15de740, 0xc04cb95aa0, 0x409f35, 0x11cf680, 0x12cc6$
        /home/travis/.gimme/versions/go1.12.4.linux.amd64/src/bytes/buffer.go:207 +0xbd
crypto/tls.(*Conn).readFromUntil(0xc000059500, 0x7f531a71f068, 0xc015be2dc0, 0x5, 0xc015be2d$
        /home/travis/.gimme/versions/go1.12.4.linux.amd64/src/crypto/tls/conn.go:783 +0xf8
crypto/tls.(*Conn).readRecordOrCCS(0xc000059500, 0x13f1c00, 0xc000059638, 0xc022f46b88)
        /home/travis/.gimme/versions/go1.12.4.linux.amd64/src/crypto/tls/conn.go:590 +0x125
crypto/tls.(*Conn).readRecord(...)
        /home/travis/.gimme/versions/go1.12.4.linux.amd64/src/bufio/bufio.go:100 +0x10f
bufio.(*Reader).Peek(0xc000077da0, 0x1, 0x0, 0x0, 0x1, 0xc04d0d4c00, 0x0)
        /home/travis/.gimme/versions/go1.12.4.linux.amd64/src/bufio/bufio.go:138 +0x4f
net/http.(*persistConn).readLoop(0xc0001f4000)
        /home/travis/.gimme/versions/go1.12.4.linux.amd64/src/net/http/transport.go:1677 +0x$
created by net/http.(*Transport).dialConn
        /home/travis/.gimme/versions/go1.12.4.linux.amd64/src/net/http/transport.go:1357 +0x$

goroutine 50 [select]:
github.com/ncw/rclone/fs/sync.(*pipe).Get(0xc00065c700, 0x1601b40, 0xc00065c7c0, 0xc01728aae$
        /home/travis/gopath/src/github.com/ncw/rclone/fs/sync/pipe.go:62 +0xe4
github.com/ncw/rclone/fs/sync.(*syncCopyMove).pairChecker(0xc0004356c0, 0xc00065c700, 0xc000$
        /home/travis/gopath/src/github.com/ncw/rclone/fs/sync/sync.go:213 +0xb0
created by github.com/ncw/rclone/fs/sync.(*syncCopyMove).startCheckers
        /home/travis/gopath/src/github.com/ncw/rclone/fs/sync/sync.go:306 +0x9a

goroutine 16 [select]:
net/http.(*persistConn).writeLoop(0xc0001f4000)
        /home/travis/.gimme/versions/go1.12.4.linux.amd64/src/net/http/transport.go:1958 +0x$
created by net/http.(*Transport).dialConn
        /home/travis/.gimme/versions/go1.12.4.linux.amd64/src/net/http/transport.go:1358 +0x$

goroutine 521 [select]:
net/http.(*persistConn).writeLoop(0xc0003d4480)
        /home/travis/.gimme/versions/go1.12.4.linux.amd64/src/net/http/transport.go:1958 +0x$
created by net/http.(*Transport).dialConn
        /home/travis/.gimme/versions/go1.12.4.linux.amd64/src/net/http/transport.go:1358 +0x$

goroutine 286 [IO wait]:
internal/poll.runtime_pollWait(0x7f531a6dbd08, 0x72, 0xffffffffffffffff)
        /home/travis/.gimme/versions/go1.12.4.linux.amd64/src/runtime/netpoll.go:182 +0x56
internal/poll.(*pollDesc).wait(0xc0000a7998, 0x72, 0x1000, 0x1000, 0xffffffffffffffff)
        /home/travis/.gimme/versions/go1.12.4.linux.amd64/src/internal/poll/fd_poll_runtime.$
internal/poll.(*pollDesc).waitRead(...)
        /home/travis/.gimme/versions/go1.12.4.linux.amd64/src/internal/poll/fd_poll_runtime.$
internal/poll.(*FD).Read(0xc0000a7980, 0xc01d482000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        /home/travis/.gimme/versions/go1.12.4.linux.amd64/src/internal/poll/fd_unix.go:169 +$
net.(*netFD).Read(0xc0000a7980, 0xc01d482000, 0x1000, 0x1000, 0x1, 0xc01a351c01, 0x100000000$
        /home/travis/.gimme/versions/go1.12.4.linux.amd64/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc0001bbcd8, 0xc01d482000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        /home/travis/.gimme/versions/go1.12.4.linux.amd64/src/net/net.go:177 +0x69
github.com/ncw/rclone/fs/fshttp.(*timeoutConn).readOrWrite(0xc0195745c0, 0xc01a351b88, 0xc01$
        /home/travis/gopath/src/github.com/ncw/rclone/fs/fshttp/http.go:75 +0x48
github.com/ncw/rclone/fs/fshttp.(*timeoutConn).Read(0xc0195745c0, 0xc01d482000, 0x1000, 0x10$
        /home/travis/gopath/src/github.com/ncw/rclone/fs/fshttp/http.go:87 +0x8a
net/http.(*persistConn).Read(0xc0001f50e0, 0xc01d482000, 0x1000, 0x1000, 0x459ff0, 0xc01a351$
        /home/travis/.gimme/versions/go1.12.4.linux.amd64/src/net/http/transport.go:1524 +0x$
bufio.(*Reader).fill(0xc01c07c720)
        /home/travis/.gimme/versions/go1.12.4.linux.amd64/src/bufio/bufio.go:100 +0x10f
bufio.(*Reader).Peek(0xc01c07c720, 0x1, 0x2, 0x0, 0x0, 0xc0017dc000, 0x0)
        /home/travis/.gimme/versions/go1.12.4.linux.amd64/src/bufio/bufio.go:138 +0x4f
net/http.(*persistConn).readLoop(0xc0001f50e0)
        /home/travis/.gimme/versions/go1.12.4.linux.amd64/src/net/http/transport.go:1677 +0x$
created by net/http.(*Transport).dialConn
        /home/travis/.gimme/versions/go1.12.4.linux.amd64/src/net/http/transport.go:1357 +0x$

goroutine 706 [IO wait]:
internal/poll.runtime_pollWait(0x7f531a6db1a8, 0x72, 0xffffffffffffffff)
        /home/travis/.gimme/versions/go1.12.4.linux.amd64/src/runtime/netpoll.go:182 +0x56
internal/poll.(*pollDesc).wait(0xc04c425e18, 0x72, 0x1000, 0x1000, 0xffffffffffffffff)
        /home/travis/.gimme/versions/go1.12.4.linux.amd64/src/internal/poll/fd_poll_runtime.$
internal/poll.(*pollDesc).waitRead(...)
        /home/travis/.gimme/versions/go1.12.4.linux.amd64/src/internal/poll/fd_poll_runtime.$
internal/poll.(*FD).Read(0xc04c425e00, 0xc02d633000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        /home/travis/.gimme/versions/go1.12.4.linux.amd64/src/internal/poll/fd_unix.go:169 +$
net.(*netFD).Read(0xc04c425e00, 0xc02d633000, 0x1000, 0x1000, 0x16, 0x11f9301, 0x38)
        /home/travis/.gimme/versions/go1.12.4.linux.amd64/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc0000a0178, 0xc02d633000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        /home/travis/.gimme/versions/go1.12.4.linux.amd64/src/net/net.go:177 +0x69
github.com/ncw/rclone/fs/fshttp.(*timeoutConn).readOrWrite(0xc000328820, 0xc04e1d0b88, 0xc02$
        /home/travis/gopath/src/github.com/ncw/rclone/fs/fshttp/http.go:75 +0x48
github.com/ncw/rclone/fs/fshttp.(*timeoutConn).Read(0xc000328820, 0xc02d633000, 0x1000, 0x10$
        /home/travis/gopath/src/github.com/ncw/rclone/fs/fshttp/http.go:87 +0x8a
net/http.(*persistConn).Read(0xc0000bcfc0, 0xc02d633000, 0x1000, 0x1000, 0xc04e1d0c88, 0x404$
        /home/travis/.gimme/versions/go1.12.4.linux.amd64/src/net/http/transport.go:1524 +0x$
bufio.(*Reader).fill(0xc0000168a0)
        /home/travis/.gimme/versions/go1.12.4.linux.amd64/src/bufio/bufio.go:100 +0x10f
bufio.(*Reader).Peek(0xc0000168a0, 0x1, 0x0, 0x0, 0x1, 0xc04f56ce00, 0x0)
        /home/travis/.gimme/versions/go1.12.4.linux.amd64/src/bufio/bufio.go:138 +0x4f
net/http.(*persistConn).readLoop(0xc0000bcfc0)
        /home/travis/.gimme/versions/go1.12.4.linux.amd64/src/net/http/transport.go:1677 +0x$
created by net/http.(*Transport).dialConn
        /home/travis/.gimme/versions/go1.12.4.linux.amd64/src/net/http/transport.go:1357 +0x$

goroutine 287 [select]:
net/http.(*persistConn).writeLoop(0xc0001f50e0)
        /home/travis/.gimme/versions/go1.12.4.linux.amd64/src/net/http/transport.go:1958 +0x$
created by net/http.(*Transport).dialConn
        /home/travis/.gimme/versions/go1.12.4.linux.amd64/src/net/http/transport.go:1358 +0x$

goroutine 55 [select]:
github.com/ncw/rclone/fs/sync.(*pipe).Get(0xc00065c700, 0x1601b40, 0xc00065c7c0, 0xc0353010e$
        /home/travis/gopath/src/github.com/ncw/rclone/fs/sync/pipe.go:62 +0xe4
github.com/ncw/rclone/fs/sync.(*syncCopyMove).pairChecker(0xc0004356c0, 0xc00065c700, 0xc000$
        /home/travis/gopath/src/github.com/ncw/rclone/fs/sync/sync.go:213 +0xb0
created by github.com/ncw/rclone/fs/sync.(*syncCopyMove).startCheckers
        /home/travis/gopath/src/github.com/ncw/rclone/fs/sync/sync.go:306 +0x9a

And a lot of similar mistakes ...
2

You can try to reduce the --buffer-size 0M, reduce the number of transfers --transfers 1 as a start.

How much memory is on the system ?

3

2Gb

bprppwN.png760×117

4

--use-mmap will help reduce memory too.
Lowest memory usage I know of might look like
rclone copy source: dest: --checkers 1 --transfers 1 --use-mmap --buffer-size 0M --tpslimit 1 --no-traverse

You can tweak from there. Technically tpslimit can go lower (0.25, for example) but I don't know how much you'll save on memory, and your transfers will take a long time.

5

rclone copy --ignore-existing -P --exclude-from "exclude-file.txt" --buffer-size 0M --transfers 1 "/home/my" MEGA:my

Transferred:              0 / 0 Bytes, -, 0 Bytes/s, ETA -
Errors:                 0
Checks:             10045 / 10045, 100%
Transferred:            0 / 0, -
Elapsed time:    46m50.3s

fatal error: runtime: out of memory
6

Is that exclude file large?

7

Thx, it's work

rclone copy --ignore-existing -P --exclude-from "exclude-file.txt" --checkers 1 --transfers 1 --use-mmap --buffer-size 0M --tpslimit 1 --no-traverse "/home/my" MEGA:my

but 344Mb transfer 6 hours

Transferred:      344.256M / 1.811 GBytes, 19%, 14.552 kBytes/s, ETA 29h31m25s
Errors:                 1 (retrying may help)
Checks:                 5 / 5, 100%
Transferred:         1758 / 11760, 15%
Elapsed time:  6h43m44.7s
Transferring:
 *                      my/full/223343.html:100% /14.068k, 52/s, 0s

now run like this

rclone copy --ignore-existing -P --exclude-from "exclude-file.txt" --use-mmap --buffer-size 0M --tpslimit 1 --no-traverse "/home/my" MEGA:my

8

It still seems to be worse now, for 7 hours only 60 megabytes.

Transferred:       65.394M / 227.661 MBytes, 29%, 2.547 kBytes/s, ETA 18h7m6s
Errors:                 1 (retrying may help)
Checks:                 6 / 6, 100%
Transferred:         4303 / 14315, 30%
Elapsed time:   7h18m6.2s
Transferring:
 *              my/sitemap/web/1203.html:100% /1, 0/s, 0s
 *              my/sitemap/web/1204.html:100% /1, 0/s, 0s
 *              my/sitemap/web/1205.html:100% /1, 0/s, 0s
 *              my/sitemap/web/1206.html:100% /1, 0/s, 0s

There is a suspicion that this is because of the encryption of mega.nz, you do not know, can it be turned off?

9

No it can't - it is part of the protocol.

The mega backend isn't as efficient as it could be

  • the upload chunks are too small
  • it keeps the whole filesystem in memory
10

New error: invalid memory address or nil pointer dereference

Does this also indicate a lack of memory?

Transferred:      188.391M / 603.611 MBytes, 31%, 2.742 kBytes/s, ETA 43h4m4s
Errors:              1360 (retrying may help)
Checks:                 6 / 6, 100%
Transferred:        11913 / 21925, 54%
Elapsed time:  19h32m25.7s
Transferring:
 *                      my/full/14048.html:100% /14.068k, 7.034k/s, 0s
 *                    my/photo/14368.html:100% /14.068k, 503/s, 0s
 *               my/image/img/14899.jpg:  0% /94.496k, 0/s, -
 * my/image/compress/…ster/small/14899.jpg:  0% /4.521k, 0/s, 
-panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0xc24b65]

goroutine 30 [running]:
github.com/ncw/rclone/vendor/github.com/t3rm1n4l/go-mega.blockDecrypt(0x0, 0x0, 0xc0080a2d20, 0x20, 0x21, 0xc0080a2d20, 0x20, 0x21, 0x0, 0x0)
        /home/travis/gopath/src/github.com/ncw/rclone/vendor/github.com/t3rm1n4l/go-mega/utils.go:202 +0x205
github.com/ncw/rclone/vendor/github.com/t3rm1n4l/go-mega.(*Mega).addFSNode(0xc000309c00, 0xc02e048fb8, 0x8, 0xc02e049160, 0x8, 0xc02e049170, 0xb, 0x0, 0xc0080a2c00, 0x2b, ...)
        /home/travis/gopath/src/github.com/ncw/rclone/vendor/github.com/t3rm1n4l/go-mega/mega.go:715 +0x327
github.com/ncw/rclone/vendor/github.com/t3rm1n4l/go-mega.(*Mega).processAddNode(0xc000309c00, 0xc0082e1a20, 0x141, 0x160, 0x0, 0x0)
        /home/travis/gopath/src/github.com/ncw/rclone/vendor/github.com/t3rm1n4l/go-mega/mega.go:1639 +0x1e6
github.com/ncw/rclone/vendor/github.com/t3rm1n4l/go-mega.(*Mega).pollEvents(0xc000309c00)
        /home/travis/gopath/src/github.com/ncw/rclone/vendor/github.com/t3rm1n4l/go-mega/mega.go:1817 +0xae0
created by github.com/ncw/rclone/vendor/github.com/t3rm1n4l/go-mega.(*Mega).getFileSystem
        /home/travis/gopath/src/github.com/ncw/rclone/vendor/github.com/t3rm1n4l/go-mega/mega.go:842 +0x3da
11

No that is a bug... I fixed a lot of those recently - can you try with the latest beta?

12

Just a note that, at least on linux, you could use something like ulimit (from bash), systemd's MemoryHigh and MemoryMax directives, or firejail's --rlimit-as option (if you're sandboxing it anyway). Just wanted to provide additional options in case those are available to you.

13

That would just cause the process to run out of memory as that's the error OP was getting. He was looking for a way to limit rclone to use x amount of memory.

14

The MemoryHigh directive limits without invoking the OOM killer, so it's what they'd want. Not sure about ulimit or --rlimit-as, though.

15

Not the OOM killer, rclone itself:

fatal error: runtime: out of memory was the error as OP is running on a low memory machine.

16

Right, so if you set MemoryHigh to, say, 50% of the total RAM, then systemd starts aggressively limiting its RAM usage once it passes that. This ensures there's enough buffer and that the OOM killer isn't invoked (as it is with MemoryMax).

17

When the process doesn't have enough memory, it will "fatal error: runtime: out of memory" which was his problem as he doesn't have enough memory on the system.

18

Hmm, I see. I guess this works as an additional check to ensure rclone can't take up all of the system's RAM, but you're right that it wouldn't be enough.

19

Setting this will make the garbage collector work harder which should improve things. The default is 100, smaller numbers make the garbage collector work harder at the expense of a bit more CPU.

export GOGC=20
20

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.