Rclone can't free the memory

What is the problem you are having with rclone?

We try to use rclone to sync a huge bucket which have about 40,000,000 objects in alibaba cloud, we run rclone in ECS that has about 32GB memory. When I monitor the memory usage, it goes up.
When about 8,000,000 are synced, the rclone is crashed. It reports out of memory issue. It seems that one object uses about 8K memory. And it will not free the memory.

What is your rclone version (output from rclone version)


Which OS you are using and how many bits (eg Windows 7, 64 bit)

Cent OS 7.3 64bit

Which cloud storage system are you using? (eg Google Drive)

Alibaba cloud OSS

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone sync

A log from the command with the -vv flag (eg output from rclone -vv copy /tmp remote:tmp)

How can rclone free the memory for the synced memory

Can you share the full command you are running?

Try the things here: https://rclone.org/faq/#rclone-is-using-too-much-memory-or-appears-to-have-a-memory-leak

Setting GOGC trades CPU usage for reduced memory usage, but it does work quite well.

rclone sync oss:zb-am54 /mnt/ds-00014fpm0z4ne13ft1m1/bp-00014fpm0z4ne1377tlq/zb-am54 --config=./rclone.conf --s3-provider=Alibaba --s3-access-key-id=LTAIHBQlJet4VzJD --s3-secret-access-key=****** --s3-endpoint=http://oss-cn-hangzhou-internal.aliyuncs.com --no-check-certificate --retries=3 --retries-sleep=5s --stats=60s --transfers=20 --log-level=INFO --log-file=./backup_2019-09-27-16:43:54_zb-am54.log -P

I am using JAVA to wrap rclone as a tool. When I add GOGC=20 and setup 4 environments as following:
Env1: 16GB memory. The previous code without GOGC setting
Env2: 16GM memory. GOGC = 20
Env3: 32GB memory. The previous code without GOGC setting
Env4: 32GB memory: GOGC = 20
Env1: The memory reaches 100% after 15 minutes.
Env2: The memory reaches 100% after 4 hours and a half.
Env3: The memory goes up smoothly. Now it is about 85%. I think it will throw OutOfMemory several hours later.
Env4: Similar to Env3

I think "GOGC" environment variable can't solve this issue. Here is the wrapper code:
List options = new ArrayList<>();
options.add(baseDirectory + "/bin/rclone");
options.add("--config=" + baseDirectory + "/conf/rclone.conf");
options.add("--s3-access-key-id=" + ak);
options.add("--s3-secret-access-key=" + sk);
options.add("--s3-endpoint=" + setDefaultHttp(endpoint));
options.add(String.format("--retries=%s", RETRIES));
options.add(String.format("--retries-sleep=%ss", RETRIES_SLEEP));
options.add(String.format("--stats=%ss", STATS));
options.add("--transfers=" + numOfThreads);
options.add("--log-level=" + rcloneLogLevel);
options.add("--log-file=" + baseDirectory + "/logs/rclone/" + "backup_" + dateFormatter.format(new Date()) + "_" + sourcePart + ".log");
ProcessBuilder processbuilder = new ProcessBuilder(options);
Map<String, String> env = processbuilder.environment();
env.put("GOGC", "20");
Process p;
try {
p = processbuilder.start();
} catch (IOException ex) {
log.error("Failed to start rclone with error {}", StringUtility.getTrace(ex));
throw ex;

Here is the stack trace:
runtime stack:
runtime.throw(0x15a528c, 0x16)
/home/travis/.gimme/versions/go1.12.9.linux.amd64/src/runtime/panic.go:617 +0x72
runtime.sysMap(0xc350000000, 0x8000000, 0x2489f18)
/home/travis/.gimme/versions/go1.12.9.linux.amd64/src/runtime/mem_linux.go:170 +0xc7
runtime.(*mheap).sysAlloc(0x2471140, 0x5284000, 0x2471150, 0x2942)
/home/travis/.gimme/versions/go1.12.9.linux.amd64/src/runtime/malloc.go:633 +0x1cd
runtime.(*mheap).grow(0x2471140, 0x2942, 0x0)
/home/travis/.gimme/versions/go1.12.9.linux.amd64/src/runtime/mheap.go:1222 +0x42
runtime.(*mheap).allocSpanLocked(0x2471140, 0x2942, 0x2489f28, 0xc09e4bdf40)
/home/travis/.gimme/versions/go1.12.9.linux.amd64/src/runtime/mheap.go:1150 +0x37f
runtime.(*mheap).alloc_m(0x2471140, 0x2942, 0x420100, 0xc07b156f48)
/home/travis/.gimme/versions/go1.12.9.linux.amd64/src/runtime/mheap.go:977 +0xc2
/home/travis/.gimme/versions/go1.12.9.linux.amd64/src/runtime/mheap.go:1048 +0x4c
runtime.(*mheap).alloc(0x2471140, 0x2942, 0x4010100, 0x100000000000200)
/home/travis/.gimme/versions/go1.12.9.linux.amd64/src/runtime/mheap.go:1047 +0x8a
runtime.largeAlloc(0x52829d0, 0x1, 0xc2574d09b8)
/home/travis/.gimme/versions/go1.12.9.linux.amd64/src/runtime/malloc.go:1055 +0x99
/home/travis/.gimme/versions/go1.12.9.linux.amd64/src/runtime/malloc.go:950 +0x46
/home/travis/.gimme/versions/go1.12.9.linux.amd64/src/runtime/asm_amd64.s:351 +0x66

goroutine 9 [running]:
/home/travis/.gimme/versions/go1.12.9.linux.amd64/src/runtime/asm_amd64.s:311 fp=0xc2574d0948 sp=0xc2574d0940 pc=0x45ac80
runtime.mallocgc(0x52829d0, 0x13f6440, 0xc10e8be501, 0x116b45b38df4)
/home/travis/.gimme/versions/go1.12.9.linux.amd64/src/runtime/malloc.go:949 +0x872 fp=0xc2574d09e8 sp=0xc2574d0948 pc=0x40c3b2
runtime.makeslice(0x13f6440, 0x1b80df, 0x1b80df, 0xc2574d0a50)
/home/travis/.gimme/versions/go1.12.9.linux.amd64/src/runtime/slice.go:49 +0x6c fp=0xc2574d0a18 sp=0xc2574d09e8 pc=0x44402c
github.com/rclone/rclone/fs/accounting.(*StatsInfo).totalDuration(0xc00013e340, 0xc00013e340)
/home/travis/gopath/src/github.com/rclone/rclone/fs/accounting/stats.go:123 +0x82 fp=0xc2574d0b10 sp=0xc2574d0a18 pc=0x7f3822
github.com/rclone/rclone/fs/accounting.(*StatsInfo).String(0xc00013e340, 0xc, 0xc00013e340)
/home/travis/gopath/src/github.com/rclone/rclone/fs/accounting/stats.go:204 +0xd6 fp=0xc2574d0e08 sp=0xc2574d0b10 pc=0x7f3fe6
github.com/rclone/rclone/cmd.printProgress(0x0, 0x0)
/home/travis/gopath/src/github.com/rclone/rclone/cmd/progress.go:96 +0xfb fp=0xc2574d0f08 sp=0xc2574d0e08 pc=0x10cb2ab
github.com/rclone/rclone/cmd.startProgress.func3(0xc0000410b0, 0xc0001b4120, 0x1616bb8)
/home/travis/gopath/src/github.com/rclone/rclone/cmd/progress.go:56 +0x9f fp=0xc2574d0fc8 sp=0xc2574d0f08 pc=0x10cc6cf
/home/travis/.gimme/versions/go1.12.9.linux.amd64/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc2574d0fd0 sp=0xc2574d0fc8 pc=0x45cbd1
created by github.com/rclone/rclone/cmd.startProgress
/home/travis/gopath/src/github.com/rclone/rclone/cmd/progress.go:46 +0xd1

I think I responded to you on the slack group, but here is a longer explanation.

Rclone buffers each directory in memory when it is transferring data. Each object from the directory uses 0.5-1k of memory, so if you have a directory with 23M objects in it, that will use up to 23GB of memory.

However rclone will try to check multiple directories at once and I think you have two very large directories, so you can help by using --checkers 1 which will cause rclone to load only one of the very large directories at once.

I have set --checkers. The parameter works well. I have two huge folders. Folder1(acesure) has more than about 10,000,000. Folder2(webRdp) has more than about 20,000,000 files. When I set --checkers to 1, I expect that when rclone finishes sync for the first folder, it will free all the memory for the first folder. Then rclone continue to sync for the second folder. Then the server only needs the memory to hold the biggest folder. But it seems it doesn't work. When rclone finish syncing for the first folder, it didn't free the memory.

Rclone probably won't return the memory to the OS but it will have it free within itself.

If you add the --rc flag you can use the tools here to investigate: https://rclone.org/rc/#debugging-memory-use

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.