Rclone 1.61.1 memory leak?

Help and Support
1

What is the problem you are having with rclone?

Every night, approximately 30GB of small files（each 200kb） are uploaded to object storage through rclone.
Rclone's memory usage increases every day until memory is full and rclone crashed.

At the beginning, rclone uses 45MB memory
[root@ecs-rmyy ~]# ps aux | grep rclone
root 1658 0.4 0.5 762236 45384 ? Sl 15:11 0:08 /usr/bin/rclone mount eos:ecloud1 /mnt/eos

One day later, uses 1.1GB
[root@ecs-rmyy ~]# ps aux | grep rclone
root 1658 6.3 15.3 2207064 1176084 ? Sl May17 165:08 /usr/bin/rclone mount eos:ecloud1 /mnt/eos
root 30832 0.0 0.0 112812 976 pts/0 S+ 10:51 0:00 grep --color=auto rclon

Until server memory is full，then rclone crashed, The mounted directory offlined.I have to restart rclone every week.

Run the command 'rclone version' and share the full output of the command.

[root@ecs-rmyy ~]# rclone version
rclone v1.61.1

  • os/version: centos 7.9.2009 (64 bit)
  • os/kernel: 3.10.0-1160.83.1.el7.x86_64 (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.19.4
  • go/linking: static
  • go/tags: none

Which cloud storage system are you using? (eg Google Drive)

s3 storage

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone mount  

Using rclone convert object storage to file directory

Below is mount parameter：
[root@ecs-rmyy ~]# cat /etc/systemd/system/mnt-eos.mount 
[Unit]
After=network-online.target
[Mount]
Type=rclone
What=eos:ecloud1
Where=/mnt/eos
Options=rw,allow_other,args2env,vfs-cache-mode=full,no-modtime,transfers=16,vfs-cache-max-age=24h,vfs-cache-max-size=2G,vfs-disk-space-total-size=80T,log-file=/temp/rclone.log,config=/root/.config/rclone/rclone.conf,cache-dir=/temp/rclone

The rclone config contents with secrets removed.

[root@ecs-rmyy ~]# cat /root/.config/rclone/rclone.conf
[eos]
type = s3
provider = ChinaMobile
access_key_id = xxxxxx
secret_access_key = xxxxxxxxxxxxxxxxxxxx
endpoint = http://xxxxxxxx
acl = private

A log from the command with the -vv flag

fatal error: runtime: out of memory

runtime stack:
runtime.throw({0x1f30814?, 0x2030?})
        runtime/panic.go:1047 +0x5d fp=0xc008439e00 sp=0xc008439dd0 pc=0x439f9d
runtime.sysMapOS(0xc1a6c00000, 0x400000?)
        runtime/mem_linux.go:187 +0x11b fp=0xc008439e48 sp=0xc008439e00 pc=0x418bdb
runtime.sysMap(0xc3ffffffff?, 0x7fd6e67f8000?, 0x42e000?)
        runtime/mem.go:142 +0x35 fp=0xc008439e78 sp=0xc008439e48 pc=0x4185b5
runtime.(*mheap).grow(0x3332b00, 0x80?)
        runtime/mheap.go:1468 +0x23d fp=0xc008439ee8 sp=0xc008439e78 pc=0x42b0bd
runtime.(*mheap).allocSpan(0x3332b00, 0x80, 0x0, 0x1)
        runtime/mheap.go:1199 +0x1be fp=0xc008439f80 sp=0xc008439ee8 pc=0x42a7fe
runtime.(*mheap).alloc.func1()
        runtime/mheap.go:918 +0x65 fp=0xc008439fc8 sp=0xc008439f80 pc=0x42a285
runtime.systemstack()
        runtime/asm_amd64.s:492 +0x49 fp=0xc008439fd0 sp=0xc008439fc8 pc=0x46c9a9

goroutine 326256797 [running]:
runtime.systemstack_switch()
        runtime/asm_amd64.s:459 fp=0xc0029eece0 sp=0xc0029eecd8 pc=0x46c940
runtime.(*mheap).alloc(0x100000?, 0x80?, 0x0?)
        runtime/mheap.go:912 +0x65 fp=0xc0029eed28 sp=0xc0029eece0 pc=0x42a1c5
runtime.(*mcache).allocLarge(0x3041314239463333?, 0x100000, 0x1)
        runtime/mcache.go:233 +0x85 fp=0xc0029eed78 sp=0xc0029eed28 pc=0x417545
runtime.mallocgc(0x100000, 0x1aaa260, 0x1)
        runtime/malloc.go:1029 +0x57e fp=0xc0029eedf0 sp=0xc0029eed78 pc=0x40d79e
runtime.makeslice(0x642e433144434436?, 0x0?, 0x0?)
        runtime/slice.go:103 +0x52 fp=0xc0029eee18 sp=0xc0029eedf0 pc=0x451d52
github.com/rclone/rclone/lib/pool.New.func1(0x4133443235393836?)
        github.com/rclone/rclone/lib/pool/pool.go:54 +0x2b fp=0xc0029eee48 sp=0xc0029eee18 pc=0x86c58b
github.com/rclone/rclone/lib/pool.(*Pool).Get(0xc00040ed20)
        github.com/rclone/rclone/lib/pool/pool.go:159 +0x83 fp=0xc0029eeed0 sp=0xc0029eee48 pc=0x86ce23
github.com/rclone/rclone/fs/asyncreader.(*AsyncReader).getBuffer(0xc0029eefa0?)
        github.com/rclone/rclone/fs/asyncreader/asyncreader.go:124 +0x56 fp=0xc0029eef18 sp=0xc0029eeed0 pc=0x86ee56
github.com/rclone/rclone/fs/asyncreader.(*AsyncReader).init.func1()
        github.com/rclone/rclone/fs/asyncreader/asyncreader.go:90 +0xff fp=0xc0029eefe0 sp=0xc0029eef18 pc=0x86ebff
runtime.goexit()
        runtime/asm_amd64.s:1594 +0x1 fp=0xc0029eefe8 sp=0xc0029eefe0 pc=0x46ea21
created by github.com/rclone/rclone/fs/asyncreader.(*AsyncReader).init
        github.com/rclone/rclone/fs/asyncreader/asyncreader.go:83 +0x1cb

goroutine 1 [select, 14929 minutes]:
runtime.gopark(0xc00071fc18?, 0x2?, 0x1?, 0x30?, 0xc00071fba4?)
        runtime/proc.go:363 +0xd6 fp=0xc000083a20 sp=0xc000083a00 pc=0x43cbb6
runtime.selectgo(0xc000083c18, 0xc00071fba0, 0x1?, 0x0, 0xc0008a58c0?, 0x1)
        runtime/select.go:328 +0x7bc fp=0xc000083b60 sp=0xc000083a20 pc=0x44ca9c
github.com/rclone/rclone/cmd/mountlib.(*MountPoint).Wait(0xc0000f6900)
        github.com/rclone/rclone/cmd/mountlib/mount.go:302 +0x27c fp=0xc000083c50 sp=0xc000083b60 pc=0x17c8d3c
github.com/rclone/rclone/cmd/mountlib.NewMountCommand.func1(0xc00040db00?, {0xc00033b040?, 0x2, 0x2?})
        github.com/rclone/rclone/cmd/mountlib/mount.go:191 +0x2d8 fp=0xc000083d20 sp=0xc000083c50 pc=0x17c82f8
github.com/spf13/cobra.(*Command).execute(0xc00040db00, {0xc00033afc0, 0x2, 0x2})
        github.com/spf13/cobra@v1.6.1/command.go:920 +0x847 fp=0xc000083e58 sp=0xc000083d20 pc=0x16ed8a7
github.com/spf13/cobra.(*Command).ExecuteC(0x32f3c40)
        github.com/spf13/cobra@v1.6.1/command.go:1044 +0x3bd fp=0xc000083f10 sp=0xc000083e58 pc=0x16ee13d
github.com/spf13/cobra.(*Command).Execute(...)
        github.com/spf13/cobra@v1.6.1/command.go:968
github.com/rclone/rclone/cmd.Main()
        github.com/rclone/rclone/cmd/cmd.go:559 +0x76 fp=0xc000083f70 sp=0xc000083f10 pc=0x1718a76
main.main()
        github.com/rclone/rclone/rclone.go:14 +0x17 fp=0xc000083f80 sp=0xc000083f70 pc=0x1947437
runtime.main()
        runtime/proc.go:250 +0x212 fp=0xc000083fe0 sp=0xc000083f80 pc=0x43c7f2
runtime.goexit()
        runtime/asm_amd64.s:1594 +0x1 fp=0xc000083fe8 sp=0xc000083fe0 pc=0x46ea21

goroutine 2 [force gc (idle), 4 minutes]:
runtime.gopark(0x3315010?, 0x32c33a3bf5102?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:363 +0xd6 fp=0xc00006efb0 sp=0xc00006ef90 pc=0x43cbb6
runtime.goparkunlock(...)
        runtime/proc.go:369
runtime.forcegchelper()
        runtime/proc.go:302 +0xad fp=0xc00006efe0 sp=0xc00006efb0 pc=0x43ca4d
runtime.goexit()
        runtime/asm_amd64.s:1594 +0x1 fp=0xc00006efe8 sp=0xc00006efe0 pc=0x46ea21
created by runtime.init.6
        runtime/proc.go:290 +0x25

goroutine 17 [GC sweep wait]:
runtime.gopark(0x3317801?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:363 +0xd6 fp=0xc00006a790 sp=0xc00006a770 pc=0x43cbb6
runtime.goparkunlock(...)
        runtime/proc.go:369
runtime.bgsweep(0x0?)
        runtime/mgcsweep.go:297 +0xd7 fp=0xc00006a7c8 sp=0xc00006a790 pc=0x4272b7
runtime.gcenable.func1()
        runtime/mgc.go:178 +0x26 fp=0xc00006a7e0 sp=0xc00006a7c8 pc=0x41bda6
runtime.goexit()
        runtime/asm_amd64.s:1594 +0x1 fp=0xc00006a7e8 sp=0xc00006a7e0 pc=0x46ea21
created by runtime.gcenable
        runtime/mgc.go:178 +0x6b

goroutine 18 [GC scavenge wait]:
runtime.gopark(0x3318560?, 0x17ffcab?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:363 +0xd6 fp=0xc00006af70 sp=0xc00006af50 pc=0x43cbb6
runtime.goparkunlock(...)
        runtime/proc.go:369
runtime.(*scavengerState).park(0x3318560)
        runtime/mgcscavenge.go:389 +0x53 fp=0xc00006afa0 sp=0xc00006af70 pc=0x425313
runtime.bgscavenge(0x0?)
        runtime/mgcscavenge.go:622 +0x65 fp=0xc00006afc8 sp=0xc00006afa0 pc=0x425905
runtime.gcenable.func2()
        runtime/mgc.go:179 +0x26 fp=0xc00006afe0 sp=0xc00006afc8 pc=0x41bd46
runtime.goexit()
        runtime/asm_amd64.s:1594 +0x1 fp=0xc00006afe8 sp=0xc00006afe0 pc=0x46ea21
created by runtime.gcenable
        runtime/mgc.go:179 +0xaa

goroutine 3 [finalizer wait, 14929 minutes]:
runtime.gopark(0x0?, 0x2062678?, 0x60?, 0xc4?, 0x2000000020?)
        runtime/proc.go:363 +0xd6 fp=0xc00006e628 sp=0xc00006e608 pc=0x43cbb6
runtime.goparkunlock(...)
        runtime/proc.go:369
runtime.runfinq()
        runtime/mfinal.go:180 +0x10f fp=0xc00006e7e0 sp=0xc00006e628 pc=0x41aeaf
runtime.goexit()
        runtime/asm_amd64.s:1594 +0x1 fp=0xc00006e7e8 sp=0xc00006e7e0 pc=0x46ea21
created by runtime.createfing
        runtime/mfinal.go:157 +0x45

goroutine 19 [GC worker (idle)]:
runtime.gopark(0x32ea65d344992?, 0x1?, 0x46?, 0x11?, 0x0?)
        runtime/proc.go:363 +0xd6 fp=0xc00006ff50 sp=0xc00006ff30 pc=0x43cbb6
runtime.gcBgMarkWorker()
        runtime/mgc.go:1235 +0xf1 fp=0xc00006ffe0 sp=0xc00006ff50 pc=0x41def1
runtime.goexit()
        runtime/asm_amd64.s:1594 +0x1 fp=0xc00006ffe8 sp=0xc00006ffe0 pc=0x46ea21
created by runtime.gcBgMarkStartWorkers
        runtime/mgc.go:1159 +0x25

goroutine 20 [GC worker (idle), 6 minutes]:
runtime.gopark(0x32e64589c3eb8?, 0x1?, 0xa5?, 0xd0?, 0x0?)
        runtime/proc.go:363 +0xd6 fp=0xc00006b750 sp=0xc00006b730 pc=0x43cbb6
runtime.gcBgMarkWorker()
        runtime/mgc.go:1235 +0xf1 fp=0xc00006b7e0 sp=0xc00006b750 pc=0x41def1
runtime.goexit()
        runtime/asm_amd64.s:1594 +0x1 fp=0xc00006b7e8 sp=0xc00006b7e0 pc=0x46ea21
created by runtime.gcBgMarkStartWorkers
        runtime/mgc.go:1159 +0x25

goroutine 33 [GC worker (idle)]:
runtime.gopark(0x334baa0?, 0x1?, 0x11?, 0xa5?, 0x0?)
        runtime/proc.go:363 +0xd6 fp=0xc00006f750 sp=0xc00006f730 pc=0x43cbb6
runtime.gcBgMarkWorker()
        runtime/mgc.go:1235 +0xf1 fp=0xc00006f7e0 sp=0xc00006f750 pc=0x41def1
runtime.goexit()
        runtime/asm_amd64.s:1594 +0x1 fp=0xc00006f7e8 sp=0xc00006f7e0 pc=0x46ea21
created by runtime.gcBgMarkStartWorkers
        runtime/mgc.go:1159 +0x25

goroutine 34 [GC worker (idle)]:
runtime.gopark(0x20628b0?, 0xc0000cd2e0?, 0x18?, 0x14?, 0x0?)
        runtime/proc.go:363 +0xd6 fp=0xc000296750 sp=0xc000296730 pc=0x43cbb6
runtime.gcBgMarkWorker()
        runtime/mgc.go:1235 +0xf1 fp=0xc0002967e0 sp=0xc000296750 pc=0x41def1
runtime.goexit()
        runtime/asm_amd64.s:1594 +0x1 fp=0xc0002967e8 sp=0xc0002967e0 pc=0x46ea21
created by runtime.gcBgMarkStartWorkers
        runtime/mgc.go:1159 +0x25

goroutine 7 [select]:
runtime.gopark(0xc00060cf88?, 0x3?, 0x3?, 0x0?, 0xc00060cf72?)
        runtime/proc.go:363 +0xd6 fp=0xc00060cdf8 sp=0xc00060cdd8 pc=0x43cbb6
runtime.selectgo(0xc00060cf88, 0xc00060cf6c, 0x0?, 0x0, 0x3?, 0x1)
        runtime/select.go:328 +0x7bc fp=0xc00060cf38 sp=0xc00060cdf8 pc=0x44ca9c
go.opencensus.io/stats/view.(*worker).start(0xc000338980)
        go.opencensus.io@v0.24.0/stats/view/worker.go:292 +0xad fp=0xc00060cfc8 sp=0xc00060cf38 pc=0xcf41ad
go.opencensus.io/stats/view.init.0.func1()
        go.opencensus.io@v0.24.0/stats/view/worker.go:34 +0x26 fp=0xc00060cfe0 sp=0xc00060cfc8 pc=0xcf33e6
runtime.goexit()
        runtime/asm_amd64.s:1594 +0x1 fp=0xc00060cfe8 sp=0xc00060cfe0 pc=0x46ea21
created by go.opencensus.io/stats/view.init.0
        go.opencensus.io@v0.24.0/stats/view/worker.go:34 +0x8d

goroutine 52 [syscall, 14929 minutes]:
runtime.notetsleepg(0x443ba5?, 0xc0000c7920?)
        runtime/lock_futex.go:236 +0x34 fp=0xc0000717a0 sp=0xc000071768 pc=0x40c3d4
os/signal.signal_recv()
        runtime/sigqueue.go:152 +0x2f fp=0xc0000717c0 sp=0xc0000717a0 pc=0x46ae6f
os/signal.loop()
        os/signal/signal_unix.go:23 +0x19 fp=0xc0000717e0 sp=0xc0000717c0 pc=0x7a59b9
runtime.goexit()
        runtime/asm_amd64.s:1594 +0x1 fp=0xc0000717e8 sp=0xc0000717e0 pc=0x46ea21
created by os/signal.Notify.func1.1
        os/signal/signal.go:151 +0x2a

goroutine 36 [chan receive, 14929 minutes]:
runtime.gopark(0xc000262ea0?, 0xc0000706d0?, 0x0?, 0x0?, 0xc000100360?)
        runtime/proc.go:363 +0xd6 fp=0xc0000706b0 sp=0xc000070690 pc=0x43cbb6
runtime.chanrecv(0xc000114fc0, 0xc0000707c0, 0x1)
        runtime/chan.go:583 +0x49b fp=0xc000070740 sp=0xc0000706b0 pc=0x406cdb
runtime.chanrecv1(0xc00089f8a0?, 0x2000100000002?)
        runtime/chan.go:442 +0x18 fp=0xc000070768 sp=0xc000070740 pc=0x4067d8
github.com/rclone/rclone/lib/atexit.Register.func1.1()
        github.com/rclone/rclone/lib/atexit/atexit.go:45 +0x2f fp=0xc0000707e0 sp=0xc000070768 pc=0x7a5a6f
runtime.goexit()
        runtime/asm_amd64.s:1594 +0x1 fp=0xc0000707e8 sp=0xc0000707e0 pc=0x46ea21
created by github.com/rclone/rclone/lib/atexit.Register.func1
        github.com/rclone/rclone/lib/atexit/atexit.go:44 +0x6b

goroutine 74 [select]:
runtime.gopark(0xc0017f9f70?, 0x3?, 0xfc?, 0xce?, 0xc0017f9f42?)
        runtime/proc.go:363 +0xd6 fp=0xc0017f9dc0 sp=0xc0017f9da0 pc=0x43cbb6
runtime.selectgo(0xc0017f9f70, 0xc0017f9f3c, 0x0?, 0x0, 0x3?, 0x1)
        runtime/select.go:328 +0x7bc fp=0xc0017f9f00 sp=0xc0017f9dc0 pc=0x44ca9c
github.com/rclone/rclone/vfs/vfscache.(*Cache).cleaner(0xc00042e960, {0x2440e88, 0xc000810900})
        github.com/rclone/rclone/vfs/vfscache/cache.go:797 +0x172 fp=0xc0017f9fb8 sp=0xc0017f9f00 pc=0x179fdf2
github.com/rclone/rclone/vfs/vfscache.New.func1()
        github.com/rclone/rclone/vfs/vfscache/cache.go:144 +0x2e fp=0xc0017f9fe0 sp=0xc0017f9fb8 pc=0x179b42e
runtime.goexit()
        runtime/asm_amd64.s:1594 +0x1 fp=0xc0017f9fe8 sp=0xc0017f9fe0 pc=0x46ea21
created by github.com/rclone/rclone/vfs/vfscache.New
        github.com/rclone/rclone/vfs/vfscache/cache.go:144 +0x6fe

goroutine 22 [syscall]:
syscall.Syscall(0x47938e?, 0x32e3da0?, 0x40dc07?, 0x1c2c8c0?)
        syscall/syscall_linux.go:68 +0x27 fp=0xc006239448 sp=0xc0062393d8 pc=0x485d87
syscall.read(0x32e3da0?, {0xc19d4d0000?, 0x24?, 0x21000?})
        syscall/zsyscall_linux_amd64.go:696 +0x45 fp=0xc006239488 sp=0xc006239448 pc=0x483905
syscall.Read(...)
        syscall/syscall_unix.go:183
bazil.org/fuse.(*Conn).ReadRequest(0xc00029dc80)
        bazil.org/fuse@v0.0.0-20200524192727-fb710f7dfd05/fuse.go:580 +0xc6 fp=0xc006239ea0 sp=0xc006239488 pc=0x1777fc6
bazil.org/fuse/fs.(*Server).Serve(0xc0006c0000, {0x242b560?, 0xc0006306c0})
        bazil.org/fuse@v0.0.0-20200524192727-fb710f7dfd05/fs/serve.go:501 +0x3ef fp=0xc006239f88 sp=0xc006239ea0 pc=0x178a00f
github.com/rclone/rclone/cmd/mount.mount.func2()
        github.com/rclone/rclone/cmd/mount/mount.go:98 +0x3e fp=0xc006239fe0 sp=0xc006239f88 pc=0x17d2f3e
runtime.goexit()
        runtime/asm_amd64.s:1594 +0x1 fp=0xc006239fe8 sp=0xc006239fe0 pc=0x46ea21
created by github.com/rclone/rclone/cmd/mount.mount
        github.com/rclone/rclone/cmd/mount/mount.go:97 +0x38a

goroutine 23 [select, 14929 minutes, locked to thread]:
runtime.gopark(0xc000608fa8?, 0x2?, 0x8e?, 0xc9?, 0xc000608fa4?)
        runtime/proc.go:363 +0xd6 fp=0xc000608e18 sp=0xc000608df8 pc=0x43cbb6
runtime.selectgo(0xc000608fa8, 0xc000608fa0, 0x0?, 0x0, 0x0?, 0x1)
        runtime/select.go:328 +0x7bc fp=0xc000608f58 sp=0xc000608e18 pc=0x44ca9c
runtime.ensureSigM.func1()
        runtime/signal_unix.go:991 +0x1b0 fp=0xc000608fe0 sp=0xc000608f58 pc=0x450ed0
runtime.goexit()
        runtime/asm_amd64.s:1594 +0x1 fp=0xc000608fe8 sp=0xc000608fe0 pc=0x46ea21
created by runtime.ensureSigM
        runtime/signal_unix.go:974 +0xbd

goroutine 326256808 [runnable]:
github.com/rclone/rclone/lib/readers.(*contextReader).Read(0xc00349bc80, {0xc003c5a000, 0x8000, 0x8000})
        github.com/rclone/rclone/lib/readers/context.go:23 +0x94 fp=0xc008821308 sp=0xc008821300 pc=0x86d4b4
io.copyBuffer({0x242d2e0, 0xc0677aa750}, {0x242b9e0, 0xc00349bc80}, {0x0, 0x0, 0x0})
        io/io.go:427 +0x1b2 fp=0xc008821388 sp=0xc008821308 pc=0x49edb2
io.Copy(...)
        io/io.go:386
github.com/rclone/rclone/fs/hash.StreamTypes({0x242b9e0, 0xc00349bc80}, 0x0?)
        github.com/rclone/rclone/fs/hash/hash.go:119 +0x2a7 fp=0xc0088214f0 sp=0xc008821388 pc=0x591ac7
github.com/rclone/rclone/backend/local.(*Object).Hash(0xc19476acc0, {0x2440e88?, 0xc190c46280}, 0x1)
        github.com/rclone/rclone/backend/local/local.go:941 +0x689 fp=0xc0088215f0 sp=0xc0088214f0 pc=0xfdd4e9
github.com/rclone/rclone/backend/s3.(*Object).Update(0xc19d490090, {0x2440e88, 0xc190c46280}, {0x242b740, 0xc006d47200}, {0x7fd6bef944f0?, 0xc19476acc0?}, {0xc1331c52c0, 0x1, 0x1})
        github.com/rclone/rclone/backend/s3/s3.go:5397 +0xd89 fp=0xc008821a50 sp=0xc0088215f0 pc=0x13de529
github.com/rclone/rclone/backend/s3.(*Fs).Put(0xc0000f9340, {0x2440e88, 0xc190c46280}, {0x242b740, 0xc006d47200}, {0x7fd6bef944f0?, 0xc19476acc0?}, {0xc1331c52c0, 0x1, 0x1})
        github.com/rclone/rclone/backend/s3/s3.go:3760 +0x125 fp=0xc008821ad8 sp=0xc008821a50 pc=0x13cfe85
github.com/rclone/rclone/fs/operations.Copy({0x2440e88, 0xc190c46280}, {0x24535c0, 0xc0000f9340}, {0x0?, 0x0?}, {0xc168f7e780, 0x74}, {0x2452bb0, 0xc19476acc0})
        github.com/rclone/rclone/fs/operations/operations.go:440 +0x1c84 fp=0xc008821d58 sp=0xc008821ad8 pc=0xae9864
github.com/rclone/rclone/vfs/vfscache.(*Item)._store(0xc006d46600, {0x2440e88, 0xc190c46280}, 0xc172eba130)
        github.com/rclone/rclone/vfs/vfscache/item.go:600 +0x1e5 fp=0xc008821e38 sp=0xc008821d58 pc=0x17a3a45
github.com/rclone/rclone/vfs/vfscache.(*Item).store(0x40b587?, {0x2440e88?, 0xc190c46280?}, 0x1?)
        github.com/rclone/rclone/vfs/vfscache/item.go:636 +0xcc fp=0xc008821eb0 sp=0xc008821e38 pc=0x17a3dec
github.com/rclone/rclone/vfs/vfscache.(*Item).Close.func2({0x2440e88?, 0xc190c46280?})
        github.com/rclone/rclone/vfs/vfscache/item.go:735 +0x34 fp=0xc008821ee0 sp=0xc008821eb0 pc=0x17a49b4
github.com/rclone/rclone/vfs/vfscache/writeback.(*WriteBack).upload(0xc000250e70, {0x2440e88, 0xc190c46280}, 0xc0038b6e70)
        github.com/rclone/rclone/vfs/vfscache/writeback/writeback.go:354 +0x137 fp=0xc008821fb0 sp=0xc008821ee0 pc=0x17969b7
github.com/rclone/rclone/vfs/vfscache/writeback.(*WriteBack).processItems.func2()
        github.com/rclone/rclone/vfs/vfscache/writeback/writeback.go:450 +0x32 fp=0xc008821fe0 sp=0xc008821fb0 pc=0x1797332
runtime.goexit()
        runtime/asm_amd64.s:1594 +0x1 fp=0xc008821fe8 sp=0xc008821fe0 pc=0x46ea21
created by github.com/rclone/rclone/vfs/vfscache/writeback.(*WriteBack).processItems
        github.com/rclone/rclone/vfs/vfscache/writeback/writeback.go:450 +0x2af

goroutine 326256811 [runnable]:
reflect.Value.Elem({0x1d1f900?, 0xc0038b7490?, 0x16?})
        reflect/value.go:1212 +0x1ba fp=0xc0029ec990 sp=0xc0029ec988 pc=0x4ca79a
reflect.Indirect(...)
        reflect/value.go:3034
bazil.org/fuse/fs.opName({0x243a390?, 0xc0038b7490?})
        bazil.org/fuse@v0.0.0-20200524192727-fb710f7dfd05/fs/serve.go:816 +0xb5 fp=0xc0029ec9d8 sp=0xc0029ec990 pc=0x178c3d5
bazil.org/fuse/fs.(*Server).serve.func1({0x1b38360, 0xc1915c21a0})
        bazil.org/fuse@v0.0.0-20200524192727-fb710f7dfd05/fs/serve.go:968 +0x5b fp=0xc0029ecae0 sp=0xc0029ec9d8 pc=0x178d3db
bazil.org/fuse/fs.(*Server).handleRequest(0xc0006c0000, {0x2440e88, 0xc1a407f840}, {0x242b580?, 0xc153d94850}, 0xc1a107b180, {0x243a390?, 0xc0038b7490?}, 0xc0029ecf08)
        bazil.org/fuse@v0.0.0-20200524192727-fb710f7dfd05/fs/serve.go:1437 +0x25d5 fp=0xc0029ecde0 sp=0xc0029ecae0 pc=0x178fc55
bazil.org/fuse/fs.(*Server).serve(0xc0006c0000, {0x243a390, 0xc0038b7490})
        bazil.org/fuse@v0.0.0-20200524192727-fb710f7dfd05/fs/serve.go:1015 +0x625 fp=0xc0029ecf98 sp=0xc0029ecde0 pc=0x178cf25
bazil.org/fuse/fs.(*Server).Serve.func1()
        bazil.org/fuse@v0.0.0-20200524192727-fb710f7dfd05/fs/serve.go:512 +0x69 fp=0xc0029ecfe0 sp=0xc0029ecf98 pc=0x178a229
runtime.goexit()
        runtime/asm_amd64.s:1594 +0x1 fp=0xc0029ecfe8 sp=0xc0029ecfe0 pc=0x46ea21
created by bazil.org/fuse/fs.(*Server).Serve
        bazil.org/fuse@v0.0.0-20200524192727-fb710f7dfd05/fs/serve.go:510 +0x3df

goroutine 326256746 [runnable]:
syscall.Syscall6(0xc0042c6260?, 0xc1a46601c0?, 0x414325?, 0x10000c0044862a0?, 0x0?, 0xc0042c6260?, 0xc02f6719e0?)
        syscall/syscall_linux.go:90 +0x36 fp=0xc00154ee90 sp=0xc00154ee08 pc=0x485e16
syscall.Syscall6(0xdd, 0xd, 0xba0000, 0x20000, 0x4, 0x0, 0x0)
        <autogenerated>:1 +0x45 fp=0xc00154eed8 sp=0xc00154ee90 pc=0x4861c5
golang.org/x/sys/unix.Fadvise(0x1f189c5?, 0xe?, 0xc0002a0d00?, 0xc00154ef70?)
        golang.org/x/sys@v0.3.0/unix/zsyscall_linux_amd64.go:67 +0x3f fp=0xc00154ef38 sp=0xc00154eed8 pc=0x5999df
github.com/rclone/rclone/backend/local.(*fadvise).worker(0xc18c9d7880)
        github.com/rclone/rclone/backend/local/fadvise_unix.go:116 +0x79 fp=0xc00154efc8 sp=0xc00154ef38 pc=0xfd8d39
github.com/rclone/rclone/backend/local.newFadvise.func1()
        github.com/rclone/rclone/backend/local/fadvise_unix.go:75 +0x26 fp=0xc00154efe0 sp=0xc00154efc8 pc=0xfd8b66
runtime.goexit()
        runtime/asm_amd64.s:1594 +0x1 fp=0xc00154efe8 sp=0xc00154efe0 pc=0x46ea21
created by github.com/rclone/rclone/backend/local.newFadvise
        github.com/rclone/rclone/backend/local/fadvise_unix.go:75 +0x12a

goroutine 326256795 [chan receive]:
runtime.gopark(0xc0029ebec8?, 0x4861c5?, 0x60?, 0x4a?, 0xc1a4660d00?)
        runtime/proc.go:363 +0xd6 fp=0xc0029ebe80 sp=0xc0029ebe60 pc=0x43cbb6
runtime.chanrecv(0xc003558000, 0xc0029ebf78, 0x1)
        runtime/chan.go:583 +0x49b fp=0xc0029ebf10 sp=0xc0029ebe80 pc=0x406cdb
runtime.chanrecv2(0xe5d511601f0a7061?, 0x9cd18f46062fc23e?)
        runtime/chan.go:447 +0x18 fp=0xc0029ebf38 sp=0xc0029ebf10 pc=0x406818
github.com/rclone/rclone/backend/local.(*fadvise).worker(0xc1a4660dc0)
        github.com/rclone/rclone/backend/local/fadvise_unix.go:115 +0x4b fp=0xc0029ebfc8 sp=0xc0029ebf38 pc=0xfd8d0b
github.com/rclone/rclone/backend/local.newFadvise.func1()
        github.com/rclone/rclone/backend/local/fadvise_unix.go:75 +0x26 fp=0xc0029ebfe0 sp=0xc0029ebfc8 pc=0xfd8b66
runtime.goexit()
        runtime/asm_amd64.s:1594 +0x1 fp=0xc0029ebfe8 sp=0xc0029ebfe0 pc=0x46ea21
created by github.com/rclone/rclone/backend/local.newFadvise
        github.com/rclone/rclone/backend/local/fadvise_unix.go:75 +0x12a

goroutine 326256796 [select]:
runtime.gopark(0xc0029edf90?, 0x2?, 0xff?, 0xff?, 0xc0029edf4c?)
        runtime/proc.go:363 +0xd6 fp=0xc0029eddb8 sp=0xc0029edd98 pc=0x43cbb6
runtime.selectgo(0xc0029edf90, 0xc0029edf48, 0x33182c0?, 0x0, 0x32eb8b2ad89e9?, 0x1)
        runtime/select.go:328 +0x7bc fp=0xc0029edef8 sp=0xc0029eddb8 pc=0x44ca9c
github.com/rclone/rclone/fs/accounting.(*Account).averageLoop(0xc006d47200)
        github.com/rclone/rclone/fs/accounting/accounting.go:212 +0x105 fp=0xc0029edfc8 sp=0xc0029edef8 pc=0x941f45
github.com/rclone/rclone/fs/accounting.newAccountSizeName.func1()
        github.com/rclone/rclone/fs/accounting/accounting.go:110 +0x26 fp=0xc0029edfe0 sp=0xc0029edfc8 pc=0x941466
runtime.goexit()
        runtime/asm_amd64.s:1594 +0x1 fp=0xc0029edfe8 sp=0xc0029edfe0 pc=0x46ea21
created by github.com/rclone/rclone/fs/accounting.newAccountSizeName
        github.com/rclone/rclone/fs/accounting/accounting.go:110 +0x456
2

That's about 150,000 new files each day?

I suspect what is happening is the directory cache is filling up. 1 file will take maybe 1k of memory to store so 150k files should take about 150M of memory which is larger than the 1.1GB you are seeing.

To dig further can you follow the instructions here to make a memory dump? Remote Control / API

You'll need to run with the --rc flag and install Go.

3

Mark, after running for over a week, the memory used by rclone has increased to 5.3GB.

[root@ecs-rmyy ~]# ps -aux | grep rclone
root      1658  9.3 72.5 7579368 5551964 ?     Sl   May17 1206:38 /usr/bin/rclone mount eos:ecloud1 /mnt/eos
root     16488  0.0  0.0 112812   980 pts/0    S+   14:52   0:00 grep --color=auto rclone

In order not to affect current applications, I prepare to create a new server to test.

1 Like
4

That would be great. I suspect we'll find the directory cache is filling up the memory.

You can try emptying the directory cache with the API if you want (provided you have the --rc flag`) using vfs/forget

5

Could you help me another question？

Downloading files without cache from rclone will list all directories where the file is located, if the directory has many files, it will cost a lot of time.

For example,Using get /fsd/25/8678735/19695261/913009102.dcm

When there is no directory cache, we can see rclone requested two directories "/fsd/25/8678735/" and "/fsd/25/8678735/19695261/"

image
image1920×279 65.5 KB

Download another file from the same directory,rclone no longer request directories:

image
image2026×170 80.4 KB

What flag should I use? thank you.