Rclone 1.61.1 memory leak?

What is the problem you are having with rclone?

Every night, approximately 30GB of small files(each 200kb) are uploaded to object storage through rclone.
Rclone's memory usage increases every day until memory is full and rclone crashed.

At the beginning, rclone uses 45MB memory
[root@ecs-rmyy ~]# ps aux | grep rclone
root 1658 0.4 0.5 762236 45384 ? Sl 15:11 0:08 /usr/bin/rclone mount eos:ecloud1 /mnt/eos

One day later, uses 1.1GB
[root@ecs-rmyy ~]# ps aux | grep rclone
root 1658 6.3 15.3 2207064 1176084 ? Sl May17 165:08 /usr/bin/rclone mount eos:ecloud1 /mnt/eos
root 30832 0.0 0.0 112812 976 pts/0 S+ 10:51 0:00 grep --color=auto rclon

Until server memory is full,then rclone crashed, The mounted directory offlined.I have to restart rclone every week.

Run the command 'rclone version' and share the full output of the command.

[root@ecs-rmyy ~]# rclone version
rclone v1.61.1

  • os/version: centos 7.9.2009 (64 bit)
  • os/kernel: 3.10.0-1160.83.1.el7.x86_64 (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.19.4
  • go/linking: static
  • go/tags: none

Which cloud storage system are you using? (eg Google Drive)

s3 storage

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone mount  

Using rclone convert object storage to file directory

Below is mount parameter:
[root@ecs-rmyy ~]# cat /etc/systemd/system/mnt-eos.mount 
[Unit]
After=network-online.target
[Mount]
Type=rclone
What=eos:ecloud1
Where=/mnt/eos
Options=rw,allow_other,args2env,vfs-cache-mode=full,no-modtime,transfers=16,vfs-cache-max-age=24h,vfs-cache-max-size=2G,vfs-disk-space-total-size=80T,log-file=/temp/rclone.log,config=/root/.config/rclone/rclone.conf,cache-dir=/temp/rclone

The rclone config contents with secrets removed.

[root@ecs-rmyy ~]# cat /root/.config/rclone/rclone.conf
[eos]
type = s3
provider = ChinaMobile
access_key_id = xxxxxx
secret_access_key = xxxxxxxxxxxxxxxxxxxx
endpoint = http://xxxxxxxx
acl = private

A log from the command with the -vv flag

fatal error: runtime: out of memory

runtime stack:
runtime.throw({0x1f30814?, 0x2030?})
        runtime/panic.go:1047 +0x5d fp=0xc008439e00 sp=0xc008439dd0 pc=0x439f9d
runtime.sysMapOS(0xc1a6c00000, 0x400000?)
        runtime/mem_linux.go:187 +0x11b fp=0xc008439e48 sp=0xc008439e00 pc=0x418bdb
runtime.sysMap(0xc3ffffffff?, 0x7fd6e67f8000?, 0x42e000?)
        runtime/mem.go:142 +0x35 fp=0xc008439e78 sp=0xc008439e48 pc=0x4185b5
runtime.(*mheap).grow(0x3332b00, 0x80?)
        runtime/mheap.go:1468 +0x23d fp=0xc008439ee8 sp=0xc008439e78 pc=0x42b0bd
runtime.(*mheap).allocSpan(0x3332b00, 0x80, 0x0, 0x1)
        runtime/mheap.go:1199 +0x1be fp=0xc008439f80 sp=0xc008439ee8 pc=0x42a7fe
runtime.(*mheap).alloc.func1()
        runtime/mheap.go:918 +0x65 fp=0xc008439fc8 sp=0xc008439f80 pc=0x42a285
runtime.systemstack()
        runtime/asm_amd64.s:492 +0x49 fp=0xc008439fd0 sp=0xc008439fc8 pc=0x46c9a9

goroutine 326256797 [running]:
runtime.systemstack_switch()
        runtime/asm_amd64.s:459 fp=0xc0029eece0 sp=0xc0029eecd8 pc=0x46c940
runtime.(*mheap).alloc(0x100000?, 0x80?, 0x0?)
        runtime/mheap.go:912 +0x65 fp=0xc0029eed28 sp=0xc0029eece0 pc=0x42a1c5
runtime.(*mcache).allocLarge(0x3041314239463333?, 0x100000, 0x1)
        runtime/mcache.go:233 +0x85 fp=0xc0029eed78 sp=0xc0029eed28 pc=0x417545
runtime.mallocgc(0x100000, 0x1aaa260, 0x1)
        runtime/malloc.go:1029 +0x57e fp=0xc0029eedf0 sp=0xc0029eed78 pc=0x40d79e
runtime.makeslice(0x642e433144434436?, 0x0?, 0x0?)
        runtime/slice.go:103 +0x52 fp=0xc0029eee18 sp=0xc0029eedf0 pc=0x451d52
github.com/rclone/rclone/lib/pool.New.func1(0x4133443235393836?)
        github.com/rclone/rclone/lib/pool/pool.go:54 +0x2b fp=0xc0029eee48 sp=0xc0029eee18 pc=0x86c58b
github.com/rclone/rclone/lib/pool.(*Pool).Get(0xc00040ed20)
        github.com/rclone/rclone/lib/pool/pool.go:159 +0x83 fp=0xc0029eeed0 sp=0xc0029eee48 pc=0x86ce23
github.com/rclone/rclone/fs/asyncreader.(*AsyncReader).getBuffer(0xc0029eefa0?)
        github.com/rclone/rclone/fs/asyncreader/asyncreader.go:124 +0x56 fp=0xc0029eef18 sp=0xc0029eeed0 pc=0x86ee56
github.com/rclone/rclone/fs/asyncreader.(*AsyncReader).init.func1()
        github.com/rclone/rclone/fs/asyncreader/asyncreader.go:90 +0xff fp=0xc0029eefe0 sp=0xc0029eef18 pc=0x86ebff
runtime.goexit()
        runtime/asm_amd64.s:1594 +0x1 fp=0xc0029eefe8 sp=0xc0029eefe0 pc=0x46ea21
created by github.com/rclone/rclone/fs/asyncreader.(*AsyncReader).init
        github.com/rclone/rclone/fs/asyncreader/asyncreader.go:83 +0x1cb

goroutine 1 [select, 14929 minutes]:
runtime.gopark(0xc00071fc18?, 0x2?, 0x1?, 0x30?, 0xc00071fba4?)
        runtime/proc.go:363 +0xd6 fp=0xc000083a20 sp=0xc000083a00 pc=0x43cbb6
runtime.selectgo(0xc000083c18, 0xc00071fba0, 0x1?, 0x0, 0xc0008a58c0?, 0x1)
        runtime/select.go:328 +0x7bc fp=0xc000083b60 sp=0xc000083a20 pc=0x44ca9c
github.com/rclone/rclone/cmd/mountlib.(*MountPoint).Wait(0xc0000f6900)
        github.com/rclone/rclone/cmd/mountlib/mount.go:302 +0x27c fp=0xc000083c50 sp=0xc000083b60 pc=0x17c8d3c
github.com/rclone/rclone/cmd/mountlib.NewMountCommand.func1(0xc00040db00?, {0xc00033b040?, 0x2, 0x2?})
        github.com/rclone/rclone/cmd/mountlib/mount.go:191 +0x2d8 fp=0xc000083d20 sp=0xc000083c50 pc=0x17c82f8
github.com/spf13/cobra.(*Command).execute(0xc00040db00, {0xc00033afc0, 0x2, 0x2})
        github.com/spf13/cobra@v1.6.1/command.go:920 +0x847 fp=0xc000083e58 sp=0xc000083d20 pc=0x16ed8a7
github.com/spf13/cobra.(*Command).ExecuteC(0x32f3c40)
        github.com/spf13/cobra@v1.6.1/command.go:1044 +0x3bd fp=0xc000083f10 sp=0xc000083e58 pc=0x16ee13d
github.com/spf13/cobra.(*Command).Execute(...)
        github.com/spf13/cobra@v1.6.1/command.go:968
github.com/rclone/rclone/cmd.Main()
        github.com/rclone/rclone/cmd/cmd.go:559 +0x76 fp=0xc000083f70 sp=0xc000083f10 pc=0x1718a76
main.main()
        github.com/rclone/rclone/rclone.go:14 +0x17 fp=0xc000083f80 sp=0xc000083f70 pc=0x1947437
runtime.main()
        runtime/proc.go:250 +0x212 fp=0xc000083fe0 sp=0xc000083f80 pc=0x43c7f2
runtime.goexit()
        runtime/asm_amd64.s:1594 +0x1 fp=0xc000083fe8 sp=0xc000083fe0 pc=0x46ea21

goroutine 2 [force gc (idle), 4 minutes]:
runtime.gopark(0x3315010?, 0x32c33a3bf5102?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:363 +0xd6 fp=0xc00006efb0 sp=0xc00006ef90 pc=0x43cbb6
runtime.goparkunlock(...)
        runtime/proc.go:369
runtime.forcegchelper()
        runtime/proc.go:302 +0xad fp=0xc00006efe0 sp=0xc00006efb0 pc=0x43ca4d
runtime.goexit()
        runtime/asm_amd64.s:1594 +0x1 fp=0xc00006efe8 sp=0xc00006efe0 pc=0x46ea21
created by runtime.init.6
        runtime/proc.go:290 +0x25

goroutine 17 [GC sweep wait]:
runtime.gopark(0x3317801?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:363 +0xd6 fp=0xc00006a790 sp=0xc00006a770 pc=0x43cbb6
runtime.goparkunlock(...)
        runtime/proc.go:369
runtime.bgsweep(0x0?)
        runtime/mgcsweep.go:297 +0xd7 fp=0xc00006a7c8 sp=0xc00006a790 pc=0x4272b7
runtime.gcenable.func1()
        runtime/mgc.go:178 +0x26 fp=0xc00006a7e0 sp=0xc00006a7c8 pc=0x41bda6
runtime.goexit()
        runtime/asm_amd64.s:1594 +0x1 fp=0xc00006a7e8 sp=0xc00006a7e0 pc=0x46ea21
created by runtime.gcenable
        runtime/mgc.go:178 +0x6b

goroutine 18 [GC scavenge wait]:
runtime.gopark(0x3318560?, 0x17ffcab?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:363 +0xd6 fp=0xc00006af70 sp=0xc00006af50 pc=0x43cbb6
runtime.goparkunlock(...)
        runtime/proc.go:369
runtime.(*scavengerState).park(0x3318560)
        runtime/mgcscavenge.go:389 +0x53 fp=0xc00006afa0 sp=0xc00006af70 pc=0x425313
runtime.bgscavenge(0x0?)
        runtime/mgcscavenge.go:622 +0x65 fp=0xc00006afc8 sp=0xc00006afa0 pc=0x425905
runtime.gcenable.func2()
        runtime/mgc.go:179 +0x26 fp=0xc00006afe0 sp=0xc00006afc8 pc=0x41bd46
runtime.goexit()
        runtime/asm_amd64.s:1594 +0x1 fp=0xc00006afe8 sp=0xc00006afe0 pc=0x46ea21
created by runtime.gcenable
        runtime/mgc.go:179 +0xaa

goroutine 3 [finalizer wait, 14929 minutes]:
runtime.gopark(0x0?, 0x2062678?, 0x60?, 0xc4?, 0x2000000020?)
        runtime/proc.go:363 +0xd6 fp=0xc00006e628 sp=0xc00006e608 pc=0x43cbb6
runtime.goparkunlock(...)
        runtime/proc.go:369
runtime.runfinq()
        runtime/mfinal.go:180 +0x10f fp=0xc00006e7e0 sp=0xc00006e628 pc=0x41aeaf
runtime.goexit()
        runtime/asm_amd64.s:1594 +0x1 fp=0xc00006e7e8 sp=0xc00006e7e0 pc=0x46ea21
created by runtime.createfing
        runtime/mfinal.go:157 +0x45

goroutine 19 [GC worker (idle)]:
runtime.gopark(0x32ea65d344992?, 0x1?, 0x46?, 0x11?, 0x0?)
        runtime/proc.go:363 +0xd6 fp=0xc00006ff50 sp=0xc00006ff30 pc=0x43cbb6
runtime.gcBgMarkWorker()
        runtime/mgc.go:1235 +0xf1 fp=0xc00006ffe0 sp=0xc00006ff50 pc=0x41def1
runtime.goexit()
        runtime/asm_amd64.s:1594 +0x1 fp=0xc00006ffe8 sp=0xc00006ffe0 pc=0x46ea21
created by runtime.gcBgMarkStartWorkers
        runtime/mgc.go:1159 +0x25

goroutine 20 [GC worker (idle), 6 minutes]:
runtime.gopark(0x32e64589c3eb8?, 0x1?, 0xa5?, 0xd0?, 0x0?)
        runtime/proc.go:363 +0xd6 fp=0xc00006b750 sp=0xc00006b730 pc=0x43cbb6
runtime.gcBgMarkWorker()
        runtime/mgc.go:1235 +0xf1 fp=0xc00006b7e0 sp=0xc00006b750 pc=0x41def1
runtime.goexit()
        runtime/asm_amd64.s:1594 +0x1 fp=0xc00006b7e8 sp=0xc00006b7e0 pc=0x46ea21
created by runtime.gcBgMarkStartWorkers
        runtime/mgc.go:1159 +0x25

goroutine 33 [GC worker (idle)]:
runtime.gopark(0x334baa0?, 0x1?, 0x11?, 0xa5?, 0x0?)
        runtime/proc.go:363 +0xd6 fp=0xc00006f750 sp=0xc00006f730 pc=0x43cbb6
runtime.gcBgMarkWorker()
        runtime/mgc.go:1235 +0xf1 fp=0xc00006f7e0 sp=0xc00006f750 pc=0x41def1
runtime.goexit()
        runtime/asm_amd64.s:1594 +0x1 fp=0xc00006f7e8 sp=0xc00006f7e0 pc=0x46ea21
created by runtime.gcBgMarkStartWorkers
        runtime/mgc.go:1159 +0x25

goroutine 34 [GC worker (idle)]:
runtime.gopark(0x20628b0?, 0xc0000cd2e0?, 0x18?, 0x14?, 0x0?)
        runtime/proc.go:363 +0xd6 fp=0xc000296750 sp=0xc000296730 pc=0x43cbb6
runtime.gcBgMarkWorker()
        runtime/mgc.go:1235 +0xf1 fp=0xc0002967e0 sp=0xc000296750 pc=0x41def1
runtime.goexit()
        runtime/asm_amd64.s:1594 +0x1 fp=0xc0002967e8 sp=0xc0002967e0 pc=0x46ea21
created by runtime.gcBgMarkStartWorkers
        runtime/mgc.go:1159 +0x25

goroutine 7 [select]:
runtime.gopark(0xc00060cf88?, 0x3?, 0x3?, 0x0?, 0xc00060cf72?)
        runtime/proc.go:363 +0xd6 fp=0xc00060cdf8 sp=0xc00060cdd8 pc=0x43cbb6
runtime.selectgo(0xc00060cf88, 0xc00060cf6c, 0x0?, 0x0, 0x3?, 0x1)
        runtime/select.go:328 +0x7bc fp=0xc00060cf38 sp=0xc00060cdf8 pc=0x44ca9c
go.opencensus.io/stats/view.(*worker).start(0xc000338980)
        go.opencensus.io@v0.24.0/stats/view/worker.go:292 +0xad fp=0xc00060cfc8 sp=0xc00060cf38 pc=0xcf41ad
go.opencensus.io/stats/view.init.0.func1()
        go.opencensus.io@v0.24.0/stats/view/worker.go:34 +0x26 fp=0xc00060cfe0 sp=0xc00060cfc8 pc=0xcf33e6
runtime.goexit()
        runtime/asm_amd64.s:1594 +0x1 fp=0xc00060cfe8 sp=0xc00060cfe0 pc=0x46ea21
created by go.opencensus.io/stats/view.init.0
        go.opencensus.io@v0.24.0/stats/view/worker.go:34 +0x8d

goroutine 52 [syscall, 14929 minutes]:
runtime.notetsleepg(0x443ba5?, 0xc0000c7920?)
        runtime/lock_futex.go:236 +0x34 fp=0xc0000717a0 sp=0xc000071768 pc=0x40c3d4
os/signal.signal_recv()
        runtime/sigqueue.go:152 +0x2f fp=0xc0000717c0 sp=0xc0000717a0 pc=0x46ae6f
os/signal.loop()
        os/signal/signal_unix.go:23 +0x19 fp=0xc0000717e0 sp=0xc0000717c0 pc=0x7a59b9
runtime.goexit()
        runtime/asm_amd64.s:1594 +0x1 fp=0xc0000717e8 sp=0xc0000717e0 pc=0x46ea21
created by os/signal.Notify.func1.1
        os/signal/signal.go:151 +0x2a

goroutine 36 [chan receive, 14929 minutes]:
runtime.gopark(0xc000262ea0?, 0xc0000706d0?, 0x0?, 0x0?, 0xc000100360?)
        runtime/proc.go:363 +0xd6 fp=0xc0000706b0 sp=0xc000070690 pc=0x43cbb6
runtime.chanrecv(0xc000114fc0, 0xc0000707c0, 0x1)
        runtime/chan.go:583 +0x49b fp=0xc000070740 sp=0xc0000706b0 pc=0x406cdb
runtime.chanrecv1(0xc00089f8a0?, 0x2000100000002?)
        runtime/chan.go:442 +0x18 fp=0xc000070768 sp=0xc000070740 pc=0x4067d8
github.com/rclone/rclone/lib/atexit.Register.func1.1()
        github.com/rclone/rclone/lib/atexit/atexit.go:45 +0x2f fp=0xc0000707e0 sp=0xc000070768 pc=0x7a5a6f
runtime.goexit()
        runtime/asm_amd64.s:1594 +0x1 fp=0xc0000707e8 sp=0xc0000707e0 pc=0x46ea21
created by github.com/rclone/rclone/lib/atexit.Register.func1
        github.com/rclone/rclone/lib/atexit/atexit.go:44 +0x6b

goroutine 74 [select]:
runtime.gopark(0xc0017f9f70?, 0x3?, 0xfc?, 0xce?, 0xc0017f9f42?)
        runtime/proc.go:363 +0xd6 fp=0xc0017f9dc0 sp=0xc0017f9da0 pc=0x43cbb6
runtime.selectgo(0xc0017f9f70, 0xc0017f9f3c, 0x0?, 0x0, 0x3?, 0x1)
        runtime/select.go:328 +0x7bc fp=0xc0017f9f00 sp=0xc0017f9dc0 pc=0x44ca9c
github.com/rclone/rclone/vfs/vfscache.(*Cache).cleaner(0xc00042e960, {0x2440e88, 0xc000810900})
        github.com/rclone/rclone/vfs/vfscache/cache.go:797 +0x172 fp=0xc0017f9fb8 sp=0xc0017f9f00 pc=0x179fdf2
github.com/rclone/rclone/vfs/vfscache.New.func1()
        github.com/rclone/rclone/vfs/vfscache/cache.go:144 +0x2e fp=0xc0017f9fe0 sp=0xc0017f9fb8 pc=0x179b42e
runtime.goexit()
        runtime/asm_amd64.s:1594 +0x1 fp=0xc0017f9fe8 sp=0xc0017f9fe0 pc=0x46ea21
created by github.com/rclone/rclone/vfs/vfscache.New
        github.com/rclone/rclone/vfs/vfscache/cache.go:144 +0x6fe

goroutine 22 [syscall]:
syscall.Syscall(0x47938e?, 0x32e3da0?, 0x40dc07?, 0x1c2c8c0?)
        syscall/syscall_linux.go:68 +0x27 fp=0xc006239448 sp=0xc0062393d8 pc=0x485d87
syscall.read(0x32e3da0?, {0xc19d4d0000?, 0x24?, 0x21000?})
        syscall/zsyscall_linux_amd64.go:696 +0x45 fp=0xc006239488 sp=0xc006239448 pc=0x483905
syscall.Read(...)
        syscall/syscall_unix.go:183
bazil.org/fuse.(*Conn).ReadRequest(0xc00029dc80)
        bazil.org/fuse@v0.0.0-20200524192727-fb710f7dfd05/fuse.go:580 +0xc6 fp=0xc006239ea0 sp=0xc006239488 pc=0x1777fc6
bazil.org/fuse/fs.(*Server).Serve(0xc0006c0000, {0x242b560?, 0xc0006306c0})
        bazil.org/fuse@v0.0.0-20200524192727-fb710f7dfd05/fs/serve.go:501 +0x3ef fp=0xc006239f88 sp=0xc006239ea0 pc=0x178a00f
github.com/rclone/rclone/cmd/mount.mount.func2()
        github.com/rclone/rclone/cmd/mount/mount.go:98 +0x3e fp=0xc006239fe0 sp=0xc006239f88 pc=0x17d2f3e
runtime.goexit()
        runtime/asm_amd64.s:1594 +0x1 fp=0xc006239fe8 sp=0xc006239fe0 pc=0x46ea21
created by github.com/rclone/rclone/cmd/mount.mount
        github.com/rclone/rclone/cmd/mount/mount.go:97 +0x38a

goroutine 23 [select, 14929 minutes, locked to thread]:
runtime.gopark(0xc000608fa8?, 0x2?, 0x8e?, 0xc9?, 0xc000608fa4?)
        runtime/proc.go:363 +0xd6 fp=0xc000608e18 sp=0xc000608df8 pc=0x43cbb6
runtime.selectgo(0xc000608fa8, 0xc000608fa0, 0x0?, 0x0, 0x0?, 0x1)
        runtime/select.go:328 +0x7bc fp=0xc000608f58 sp=0xc000608e18 pc=0x44ca9c
runtime.ensureSigM.func1()
        runtime/signal_unix.go:991 +0x1b0 fp=0xc000608fe0 sp=0xc000608f58 pc=0x450ed0
runtime.goexit()
        runtime/asm_amd64.s:1594 +0x1 fp=0xc000608fe8 sp=0xc000608fe0 pc=0x46ea21
created by runtime.ensureSigM
        runtime/signal_unix.go:974 +0xbd

goroutine 326256808 [runnable]:
github.com/rclone/rclone/lib/readers.(*contextReader).Read(0xc00349bc80, {0xc003c5a000, 0x8000, 0x8000})
        github.com/rclone/rclone/lib/readers/context.go:23 +0x94 fp=0xc008821308 sp=0xc008821300 pc=0x86d4b4
io.copyBuffer({0x242d2e0, 0xc0677aa750}, {0x242b9e0, 0xc00349bc80}, {0x0, 0x0, 0x0})
        io/io.go:427 +0x1b2 fp=0xc008821388 sp=0xc008821308 pc=0x49edb2
io.Copy(...)
        io/io.go:386
github.com/rclone/rclone/fs/hash.StreamTypes({0x242b9e0, 0xc00349bc80}, 0x0?)
        github.com/rclone/rclone/fs/hash/hash.go:119 +0x2a7 fp=0xc0088214f0 sp=0xc008821388 pc=0x591ac7
github.com/rclone/rclone/backend/local.(*Object).Hash(0xc19476acc0, {0x2440e88?, 0xc190c46280}, 0x1)
        github.com/rclone/rclone/backend/local/local.go:941 +0x689 fp=0xc0088215f0 sp=0xc0088214f0 pc=0xfdd4e9
github.com/rclone/rclone/backend/s3.(*Object).Update(0xc19d490090, {0x2440e88, 0xc190c46280}, {0x242b740, 0xc006d47200}, {0x7fd6bef944f0?, 0xc19476acc0?}, {0xc1331c52c0, 0x1, 0x1})
        github.com/rclone/rclone/backend/s3/s3.go:5397 +0xd89 fp=0xc008821a50 sp=0xc0088215f0 pc=0x13de529
github.com/rclone/rclone/backend/s3.(*Fs).Put(0xc0000f9340, {0x2440e88, 0xc190c46280}, {0x242b740, 0xc006d47200}, {0x7fd6bef944f0?, 0xc19476acc0?}, {0xc1331c52c0, 0x1, 0x1})
        github.com/rclone/rclone/backend/s3/s3.go:3760 +0x125 fp=0xc008821ad8 sp=0xc008821a50 pc=0x13cfe85
github.com/rclone/rclone/fs/operations.Copy({0x2440e88, 0xc190c46280}, {0x24535c0, 0xc0000f9340}, {0x0?, 0x0?}, {0xc168f7e780, 0x74}, {0x2452bb0, 0xc19476acc0})
        github.com/rclone/rclone/fs/operations/operations.go:440 +0x1c84 fp=0xc008821d58 sp=0xc008821ad8 pc=0xae9864
github.com/rclone/rclone/vfs/vfscache.(*Item)._store(0xc006d46600, {0x2440e88, 0xc190c46280}, 0xc172eba130)
        github.com/rclone/rclone/vfs/vfscache/item.go:600 +0x1e5 fp=0xc008821e38 sp=0xc008821d58 pc=0x17a3a45
github.com/rclone/rclone/vfs/vfscache.(*Item).store(0x40b587?, {0x2440e88?, 0xc190c46280?}, 0x1?)
        github.com/rclone/rclone/vfs/vfscache/item.go:636 +0xcc fp=0xc008821eb0 sp=0xc008821e38 pc=0x17a3dec
github.com/rclone/rclone/vfs/vfscache.(*Item).Close.func2({0x2440e88?, 0xc190c46280?})
        github.com/rclone/rclone/vfs/vfscache/item.go:735 +0x34 fp=0xc008821ee0 sp=0xc008821eb0 pc=0x17a49b4
github.com/rclone/rclone/vfs/vfscache/writeback.(*WriteBack).upload(0xc000250e70, {0x2440e88, 0xc190c46280}, 0xc0038b6e70)
        github.com/rclone/rclone/vfs/vfscache/writeback/writeback.go:354 +0x137 fp=0xc008821fb0 sp=0xc008821ee0 pc=0x17969b7
github.com/rclone/rclone/vfs/vfscache/writeback.(*WriteBack).processItems.func2()
        github.com/rclone/rclone/vfs/vfscache/writeback/writeback.go:450 +0x32 fp=0xc008821fe0 sp=0xc008821fb0 pc=0x1797332
runtime.goexit()
        runtime/asm_amd64.s:1594 +0x1 fp=0xc008821fe8 sp=0xc008821fe0 pc=0x46ea21
created by github.com/rclone/rclone/vfs/vfscache/writeback.(*WriteBack).processItems
        github.com/rclone/rclone/vfs/vfscache/writeback/writeback.go:450 +0x2af

goroutine 326256811 [runnable]:
reflect.Value.Elem({0x1d1f900?, 0xc0038b7490?, 0x16?})
        reflect/value.go:1212 +0x1ba fp=0xc0029ec990 sp=0xc0029ec988 pc=0x4ca79a
reflect.Indirect(...)
        reflect/value.go:3034
bazil.org/fuse/fs.opName({0x243a390?, 0xc0038b7490?})
        bazil.org/fuse@v0.0.0-20200524192727-fb710f7dfd05/fs/serve.go:816 +0xb5 fp=0xc0029ec9d8 sp=0xc0029ec990 pc=0x178c3d5
bazil.org/fuse/fs.(*Server).serve.func1({0x1b38360, 0xc1915c21a0})
        bazil.org/fuse@v0.0.0-20200524192727-fb710f7dfd05/fs/serve.go:968 +0x5b fp=0xc0029ecae0 sp=0xc0029ec9d8 pc=0x178d3db
bazil.org/fuse/fs.(*Server).handleRequest(0xc0006c0000, {0x2440e88, 0xc1a407f840}, {0x242b580?, 0xc153d94850}, 0xc1a107b180, {0x243a390?, 0xc0038b7490?}, 0xc0029ecf08)
        bazil.org/fuse@v0.0.0-20200524192727-fb710f7dfd05/fs/serve.go:1437 +0x25d5 fp=0xc0029ecde0 sp=0xc0029ecae0 pc=0x178fc55
bazil.org/fuse/fs.(*Server).serve(0xc0006c0000, {0x243a390, 0xc0038b7490})
        bazil.org/fuse@v0.0.0-20200524192727-fb710f7dfd05/fs/serve.go:1015 +0x625 fp=0xc0029ecf98 sp=0xc0029ecde0 pc=0x178cf25
bazil.org/fuse/fs.(*Server).Serve.func1()
        bazil.org/fuse@v0.0.0-20200524192727-fb710f7dfd05/fs/serve.go:512 +0x69 fp=0xc0029ecfe0 sp=0xc0029ecf98 pc=0x178a229
runtime.goexit()
        runtime/asm_amd64.s:1594 +0x1 fp=0xc0029ecfe8 sp=0xc0029ecfe0 pc=0x46ea21
created by bazil.org/fuse/fs.(*Server).Serve
        bazil.org/fuse@v0.0.0-20200524192727-fb710f7dfd05/fs/serve.go:510 +0x3df

goroutine 326256746 [runnable]:
syscall.Syscall6(0xc0042c6260?, 0xc1a46601c0?, 0x414325?, 0x10000c0044862a0?, 0x0?, 0xc0042c6260?, 0xc02f6719e0?)
        syscall/syscall_linux.go:90 +0x36 fp=0xc00154ee90 sp=0xc00154ee08 pc=0x485e16
syscall.Syscall6(0xdd, 0xd, 0xba0000, 0x20000, 0x4, 0x0, 0x0)
        <autogenerated>:1 +0x45 fp=0xc00154eed8 sp=0xc00154ee90 pc=0x4861c5
golang.org/x/sys/unix.Fadvise(0x1f189c5?, 0xe?, 0xc0002a0d00?, 0xc00154ef70?)
        golang.org/x/sys@v0.3.0/unix/zsyscall_linux_amd64.go:67 +0x3f fp=0xc00154ef38 sp=0xc00154eed8 pc=0x5999df
github.com/rclone/rclone/backend/local.(*fadvise).worker(0xc18c9d7880)
        github.com/rclone/rclone/backend/local/fadvise_unix.go:116 +0x79 fp=0xc00154efc8 sp=0xc00154ef38 pc=0xfd8d39
github.com/rclone/rclone/backend/local.newFadvise.func1()
        github.com/rclone/rclone/backend/local/fadvise_unix.go:75 +0x26 fp=0xc00154efe0 sp=0xc00154efc8 pc=0xfd8b66
runtime.goexit()
        runtime/asm_amd64.s:1594 +0x1 fp=0xc00154efe8 sp=0xc00154efe0 pc=0x46ea21
created by github.com/rclone/rclone/backend/local.newFadvise
        github.com/rclone/rclone/backend/local/fadvise_unix.go:75 +0x12a

goroutine 326256795 [chan receive]:
runtime.gopark(0xc0029ebec8?, 0x4861c5?, 0x60?, 0x4a?, 0xc1a4660d00?)
        runtime/proc.go:363 +0xd6 fp=0xc0029ebe80 sp=0xc0029ebe60 pc=0x43cbb6
runtime.chanrecv(0xc003558000, 0xc0029ebf78, 0x1)
        runtime/chan.go:583 +0x49b fp=0xc0029ebf10 sp=0xc0029ebe80 pc=0x406cdb
runtime.chanrecv2(0xe5d511601f0a7061?, 0x9cd18f46062fc23e?)
        runtime/chan.go:447 +0x18 fp=0xc0029ebf38 sp=0xc0029ebf10 pc=0x406818
github.com/rclone/rclone/backend/local.(*fadvise).worker(0xc1a4660dc0)
        github.com/rclone/rclone/backend/local/fadvise_unix.go:115 +0x4b fp=0xc0029ebfc8 sp=0xc0029ebf38 pc=0xfd8d0b
github.com/rclone/rclone/backend/local.newFadvise.func1()
        github.com/rclone/rclone/backend/local/fadvise_unix.go:75 +0x26 fp=0xc0029ebfe0 sp=0xc0029ebfc8 pc=0xfd8b66
runtime.goexit()
        runtime/asm_amd64.s:1594 +0x1 fp=0xc0029ebfe8 sp=0xc0029ebfe0 pc=0x46ea21
created by github.com/rclone/rclone/backend/local.newFadvise
        github.com/rclone/rclone/backend/local/fadvise_unix.go:75 +0x12a

goroutine 326256796 [select]:
runtime.gopark(0xc0029edf90?, 0x2?, 0xff?, 0xff?, 0xc0029edf4c?)
        runtime/proc.go:363 +0xd6 fp=0xc0029eddb8 sp=0xc0029edd98 pc=0x43cbb6
runtime.selectgo(0xc0029edf90, 0xc0029edf48, 0x33182c0?, 0x0, 0x32eb8b2ad89e9?, 0x1)
        runtime/select.go:328 +0x7bc fp=0xc0029edef8 sp=0xc0029eddb8 pc=0x44ca9c
github.com/rclone/rclone/fs/accounting.(*Account).averageLoop(0xc006d47200)
        github.com/rclone/rclone/fs/accounting/accounting.go:212 +0x105 fp=0xc0029edfc8 sp=0xc0029edef8 pc=0x941f45
github.com/rclone/rclone/fs/accounting.newAccountSizeName.func1()
        github.com/rclone/rclone/fs/accounting/accounting.go:110 +0x26 fp=0xc0029edfe0 sp=0xc0029edfc8 pc=0x941466
runtime.goexit()
        runtime/asm_amd64.s:1594 +0x1 fp=0xc0029edfe8 sp=0xc0029edfe0 pc=0x46ea21
created by github.com/rclone/rclone/fs/accounting.newAccountSizeName
        github.com/rclone/rclone/fs/accounting/accounting.go:110 +0x456

That's about 150,000 new files each day?

I suspect what is happening is the directory cache is filling up. 1 file will take maybe 1k of memory to store so 150k files should take about 150M of memory which is larger than the 1.1GB you are seeing.

To dig further can you follow the instructions here to make a memory dump? Remote Control / API

You'll need to run with the --rc flag and install Go.

Mark, after running for over a week, the memory used by rclone has increased to 5.3GB.

[root@ecs-rmyy ~]# ps -aux | grep rclone
root      1658  9.3 72.5 7579368 5551964 ?     Sl   May17 1206:38 /usr/bin/rclone mount eos:ecloud1 /mnt/eos
root     16488  0.0  0.0 112812   980 pts/0    S+   14:52   0:00 grep --color=auto rclone

In order not to affect current applications, I prepare to create a new server to test.

1 Like

That would be great. I suspect we'll find the directory cache is filling up the memory.

You can try emptying the directory cache with the API if you want (provided you have the --rc flag`) using vfs/forget

Could you help me another question?

Downloading files without cache from rclone will list all directories where the file is located, if the directory has many files, it will cost a lot of time.

For example,Using get /fsd/25/8678735/19695261/913009102.dcm

When there is no directory cache, we can see rclone requested two directories "/fsd/25/8678735/" and "/fsd/25/8678735/19695261/"

Download another file from the same directory,rclone no longer request directories:

What flag should I use? thank you.

Run test result:
At the beginning

[root@ecs-rclone ~]# go tool pprof -text http://localhost:5572/debug/pprof/heap
Fetching profile over HTTP from http://localhost:5572/debug/pprof/heap
Saved profile in /root/pprof/pprof.rclone.alloc_objects.alloc_space.inuse_objects.inuse_space.014.pb.gz
File: rclone
Type: inuse_space
Time: Jun 9, 2023 at 2:40pm (CST)
Showing nodes accounting for 7792.75kB, 100% of 7792.75kB total
      flat  flat%   sum%        cum   cum%
 2048.29kB 26.28% 26.28%  2048.29kB 26.28%  github.com/aws/aws-sdk-go/aws/endpoints.init
  583.01kB  7.48% 33.77%   583.01kB  7.48%  bazil.org/fuse.allocMessage
  544.67kB  6.99% 40.76%   544.67kB  6.99%  regexp/syntax.(*compiler).inst (inline)
     515kB  6.61% 47.36%      515kB  6.61%  google.golang.org/protobuf/internal/filedesc.(*File).initDecls (inline)
  514.63kB  6.60% 53.97%   514.63kB  6.60%  math/rand.NewSource (inline)
  513.50kB  6.59% 60.56%   513.50kB  6.59%  github.com/gdamore/tcell/v2/terminfo/f/foot.init.0
     513kB  6.58% 67.14%      513kB  6.58%  github.com/gdamore/encoding.init.0
  512.34kB  6.57% 73.72%   512.34kB  6.57%  github.com/colinmarc/hdfs/v2/internal/protocol/hadoop_common.init
  512.20kB  6.57% 80.29%   512.20kB  6.57%  runtime.malg
  512.05kB  6.57% 86.86%   512.05kB  6.57%  github.com/gabriel-vasile/mimetype.newMIME (inline)
  512.03kB  6.57% 93.43%   512.03kB  6.57%  text/template.New (inline)
  512.02kB  6.57%   100%   512.02kB  6.57%  math/big.nat.make (inline)
         0     0%   100%   583.01kB  7.48%  bazil.org/fuse.(*Conn).ReadRequest
         0     0%   100%   583.01kB  7.48%  bazil.org/fuse.getMessage (inline)
         0     0%   100%   583.01kB  7.48%  bazil.org/fuse/fs.(*Server).Serve
         0     0%   100%   512.02kB  6.57%  github.com/ProtonMail/go-crypto/brainpool.P256r1 (inline)
         0     0%   100%   512.02kB  6.57%  github.com/ProtonMail/go-crypto/brainpool.initAll
         0     0%   100%   512.02kB  6.57%  github.com/ProtonMail/go-crypto/brainpool.initP256r1
         0     0%   100%   512.02kB  6.57%  github.com/ProtonMail/go-crypto/openpgp/internal/ecc.init
         0     0%   100%   512.05kB  6.57%  github.com/gabriel-vasile/mimetype.init
         0     0%   100%   512.03kB  6.57%  github.com/rclone/rclone/cmd/gendocs.init
         0     0%   100%   583.01kB  7.48%  github.com/rclone/rclone/cmd/mount.mount.func2
         0     0%   100%   544.67kB  6.99%  go.opencensus.io/resource.init
         0     0%   100%   514.63kB  6.60%  golang.org/x/crypto/ssh.init
         0     0%   100%      515kB  6.61%  google.golang.org/protobuf/internal/filedesc.Builder.Build
         0     0%   100%      515kB  6.61%  google.golang.org/protobuf/internal/filedesc.newRawFile
         0     0%   100%      515kB  6.61%  google.golang.org/protobuf/internal/filetype.Builder.Build
         0     0%   100%      515kB  6.61%  google.golang.org/protobuf/types/descriptorpb.file_google_protobuf_descriptor_proto_init
         0     0%   100%      515kB  6.61%  google.golang.org/protobuf/types/descriptorpb.init.0
         0     0%   100%   512.02kB  6.57%  math/big.(*Int).SetString
         0     0%   100%   512.02kB  6.57%  math/big.(*Int).scan
         0     0%   100%   512.02kB  6.57%  math/big.(*Int).setFromScanner
         0     0%   100%   512.02kB  6.57%  math/big.nat.mulAddWW
         0     0%   100%   512.02kB  6.57%  math/big.nat.scan
         0     0%   100%   544.67kB  6.99%  regexp.Compile (inline)
         0     0%   100%   544.67kB  6.99%  regexp.MustCompile
         0     0%   100%   544.67kB  6.99%  regexp.compile
         0     0%   100%   544.67kB  6.99%  regexp/syntax.(*compiler).compile
         0     0%   100%   544.67kB  6.99%  regexp/syntax.(*compiler).quest
         0     0%   100%   544.67kB  6.99%  regexp/syntax.Compile
         0     0%   100%  6697.54kB 85.95%  runtime.doInit
         0     0%   100%  6152.87kB 78.96%  runtime.main
         0     0%   100%   512.20kB  6.57%  runtime.newproc.func1
         0     0%   100%   512.20kB  6.57%  runtime.newproc1
         0     0%   100%   512.20kB  6.57%  runtime.systemstack
         0     0%   100%   512.02kB  6.57%  sync.(*Once).Do (inline)
         0     0%   100%   512.02kB  6.57%  sync.(*Once).doSlow
         0     0%   100%   583.01kB  7.48%  sync.(*Pool).Get

After uploaded 12507files,152directories:

[root@ecs-rclone ~]# go tool pprof -text http://localhost:5572/debug/pprof/heap
Fetching profile over HTTP from http://localhost:5572/debug/pprof/heap
Saved profile in /root/pprof/pprof.rclone.alloc_objects.alloc_space.inuse_objects.inuse_space.024.pb.gz
File: rclone
Type: inuse_space
Time: Jun 9, 2023 at 3:08pm (CST)
Showing nodes accounting for 23474.76kB, 100% of 23474.76kB total
      flat  flat%   sum%        cum   cum%
 4097.01kB 17.45% 17.45%  4097.01kB 17.45%  github.com/rclone/rclone/backend/s3.s3MetadataToMap
 2560.35kB 10.91% 28.36%  6657.36kB 28.36%  github.com/rclone/rclone/backend/s3.(*Fs).Put
 2048.38kB  8.73% 37.09%  2048.38kB  8.73%  github.com/rclone/rclone/vfs.newFile
 2048.29kB  8.73% 45.81%  2048.29kB  8.73%  github.com/aws/aws-sdk-go/aws/endpoints.init
 1078.34kB  4.59% 50.40%  1078.34kB  4.59%  bazil.org/fuse/fs.(*Server).saveNode
 1024.09kB  4.36% 54.77%  1024.09kB  4.36%  strings.(*Builder).Write
 1024.06kB  4.36% 59.13%  1545.43kB  6.58%  github.com/rclone/rclone/vfs.newRWFileHandle
  768.26kB  3.27% 62.40%   768.26kB  3.27%  github.com/rclone/rclone/vfs/vfscache.(*Cache)._get
  583.01kB  2.48% 64.89%   583.01kB  2.48%  bazil.org/fuse.allocMessage
  544.67kB  2.32% 67.21%   544.67kB  2.32%  regexp/syntax.(*compiler).inst (inline)
  521.37kB  2.22% 69.43%   521.37kB  2.22%  github.com/rclone/rclone/vfs.(*Dir).addObject
     515kB  2.19% 71.62%      515kB  2.19%  google.golang.org/protobuf/internal/filedesc.(*File).initDecls (inline)
  514.63kB  2.19% 73.81%   514.63kB  2.19%  math/rand.NewSource (inline)
  513.50kB  2.19% 76.00%   513.50kB  2.19%  github.com/gdamore/tcell/v2/terminfo/f/foot.init.0
     513kB  2.19% 78.19%      513kB  2.19%  github.com/gdamore/encoding.init.0
  512.34kB  2.18% 80.37%   512.34kB  2.18%  github.com/colinmarc/hdfs/v2/internal/protocol/hadoop_common.init
  512.20kB  2.18% 82.55%   512.20kB  2.18%  runtime.malg
  512.07kB  2.18% 84.73%   512.07kB  2.18%  github.com/rclone/rclone/backend/s3.(*Fs).newObjectWithInfo
  512.05kB  2.18% 86.91%   512.05kB  2.18%  github.com/gabriel-vasile/mimetype.newMIME (inline)
  512.05kB  2.18% 89.09%   512.05kB  2.18%  github.com/rclone/rclone/fs/accounting.newAccountSizeName
  512.03kB  2.18% 91.28%   512.03kB  2.18%  text/template.New (inline)
  512.02kB  2.18% 93.46%   512.02kB  2.18%  math/big.nat.make (inline)
  512.02kB  2.18% 95.64%   512.02kB  2.18%  path.Join
  512.01kB  2.18% 97.82%   512.01kB  2.18%  net/textproto.(*Reader).ReadMIMEHeader
  512.01kB  2.18%   100%  1095.01kB  4.66%  bazil.org/fuse.(*Conn).ReadRequest
         0     0%   100%   583.01kB  2.48%  bazil.org/fuse.getMessage (inline)
         0     0%   100%  1095.01kB  4.66%  bazil.org/fuse/fs.(*Server).Serve
         0     0%   100%  6464.49kB 27.54%  bazil.org/fuse/fs.(*Server).Serve.func1
         0     0%   100%  6464.49kB 27.54%  bazil.org/fuse/fs.(*Server).handleRequest
         0     0%   100%  1078.34kB  4.59%  bazil.org/fuse/fs.(*Server).saveLookup
         0     0%   100%  6464.49kB 27.54%  bazil.org/fuse/fs.(*Server).serve
         0     0%   100%  1024.09kB  4.36%  fmt.Fprintf
         0     0%   100%   512.02kB  2.18%  github.com/ProtonMail/go-crypto/brainpool.P256r1 (inline)
         0     0%   100%   512.02kB  2.18%  github.com/ProtonMail/go-crypto/brainpool.initAll
         0     0%   100%   512.02kB  2.18%  github.com/ProtonMail/go-crypto/brainpool.initP256r1
         0     0%   100%   512.02kB  2.18%  github.com/ProtonMail/go-crypto/openpgp/internal/ecc.init
         0     0%   100%   512.05kB  2.18%  github.com/gabriel-vasile/mimetype.init
         0     0%   100%   512.07kB  2.18%  github.com/rclone/rclone/backend/s3.(*Fs).List
         0     0%   100%   512.07kB  2.18%  github.com/rclone/rclone/backend/s3.(*Fs).itemToDirEntry
         0     0%   100%   512.07kB  2.18%  github.com/rclone/rclone/backend/s3.(*Fs).list
         0     0%   100%   512.07kB  2.18%  github.com/rclone/rclone/backend/s3.(*Fs).listDir
         0     0%   100%   512.07kB  2.18%  github.com/rclone/rclone/backend/s3.(*Fs).listDir.func1
         0     0%   100%  4097.01kB 17.45%  github.com/rclone/rclone/backend/s3.(*Object).Update
         0     0%   100%  4097.01kB 17.45%  github.com/rclone/rclone/backend/s3.(*Object).setMetaData
         0     0%   100%   512.03kB  2.18%  github.com/rclone/rclone/cmd/gendocs.init
         0     0%   100%  4874.09kB 20.76%  github.com/rclone/rclone/cmd/mount.(*Dir).Create
         0     0%   100%   512.07kB  2.18%  github.com/rclone/rclone/cmd/mount.(*Dir).ReadDirAll
         0     0%   100%  1095.01kB  4.66%  github.com/rclone/rclone/cmd/mount.mount.func2
         0     0%   100%  1024.09kB  4.36%  github.com/rclone/rclone/fs.Fingerprint
         0     0%   100%   512.05kB  2.18%  github.com/rclone/rclone/fs/accounting.(*Transfer).Account
         0     0%   100%   512.07kB  2.18%  github.com/rclone/rclone/fs/list.DirSorted
         0     0%   100%  7169.41kB 30.54%  github.com/rclone/rclone/fs/operations.Copy
         0     0%   100%  2048.38kB  8.73%  github.com/rclone/rclone/vfs.(*Dir).Create
         0     0%   100%   512.07kB  2.18%  github.com/rclone/rclone/vfs.(*Dir).ReadDirAll
         0     0%   100%   512.07kB  2.18%  github.com/rclone/rclone/vfs.(*Dir)._readDir
         0     0%   100%  2825.71kB 12.04%  github.com/rclone/rclone/vfs.(*File).Open
         0     0%   100%   512.02kB  2.18%  github.com/rclone/rclone/vfs.(*File).Path
         0     0%   100%  1545.43kB  6.58%  github.com/rclone/rclone/vfs.(*File).openRW
         0     0%   100%   521.37kB  2.22%  github.com/rclone/rclone/vfs.(*RWFileHandle).Truncate
         0     0%   100%   521.37kB  2.22%  github.com/rclone/rclone/vfs.(*RWFileHandle).openPending
         0     0%   100%   768.26kB  3.27%  github.com/rclone/rclone/vfs/vfscache.(*Cache).Exists
         0     0%   100%   768.26kB  3.27%  github.com/rclone/rclone/vfs/vfscache.(*Cache).get
         0     0%   100%  8193.50kB 34.90%  github.com/rclone/rclone/vfs/vfscache.(*Item).Close.func2
         0     0%   100%  8193.50kB 34.90%  github.com/rclone/rclone/vfs/vfscache.(*Item)._store
         0     0%   100%  1024.09kB  4.36%  github.com/rclone/rclone/vfs/vfscache.(*Item)._updateFingerprint
         0     0%   100%  8193.50kB 34.90%  github.com/rclone/rclone/vfs/vfscache.(*Item).store
         0     0%   100%  8193.50kB 34.90%  github.com/rclone/rclone/vfs/vfscache/writeback.(*WriteBack).upload
         0     0%   100%   544.67kB  2.32%  go.opencensus.io/resource.init
         0     0%   100%   514.63kB  2.19%  golang.org/x/crypto/ssh.init
         0     0%   100%      515kB  2.19%  google.golang.org/protobuf/internal/filedesc.Builder.Build
         0     0%   100%      515kB  2.19%  google.golang.org/protobuf/internal/filedesc.newRawFile
         0     0%   100%      515kB  2.19%  google.golang.org/protobuf/internal/filetype.Builder.Build
         0     0%   100%      515kB  2.19%  google.golang.org/protobuf/types/descriptorpb.file_google_protobuf_descriptor_proto_init
         0     0%   100%      515kB  2.19%  google.golang.org/protobuf/types/descriptorpb.init.0
         0     0%   100%   512.02kB  2.18%  math/big.(*Int).SetString
         0     0%   100%   512.02kB  2.18%  math/big.(*Int).scan
         0     0%   100%   512.02kB  2.18%  math/big.(*Int).setFromScanner
         0     0%   100%   512.02kB  2.18%  math/big.nat.mulAddWW
         0     0%   100%   512.02kB  2.18%  math/big.nat.scan
         0     0%   100%   512.01kB  2.18%  net/http.(*persistConn).readLoop
         0     0%   100%   512.01kB  2.18%  net/http.(*persistConn).readResponse
         0     0%   100%   512.01kB  2.18%  net/http.ReadResponse
         0     0%   100%   544.67kB  2.32%  regexp.Compile (inline)
         0     0%   100%   544.67kB  2.32%  regexp.MustCompile
         0     0%   100%   544.67kB  2.32%  regexp.compile
         0     0%   100%   544.67kB  2.32%  regexp/syntax.(*compiler).compile
         0     0%   100%   544.67kB  2.32%  regexp/syntax.(*compiler).quest
         0     0%   100%   544.67kB  2.32%  regexp/syntax.Compile
         0     0%   100%  6697.54kB 28.53%  runtime.doInit
         0     0%   100%  6152.87kB 26.21%  runtime.main
         0     0%   100%   512.20kB  2.18%  runtime.newproc.func1
         0     0%   100%   512.20kB  2.18%  runtime.newproc1
         0     0%   100%   512.20kB  2.18%  runtime.systemstack
         0     0%   100%   512.02kB  2.18%  sync.(*Once).Do (inline)
         0     0%   100%   512.02kB  2.18%  sync.(*Once).doSlow
         0     0%   100%   583.01kB  2.48%  sync.(*Pool).Get

It seems not directory cache, I ran rclone rc vfs/forget, there was only a slight change in memory usage:

[root@ecs-rclone ~]# rclone rc vfs/forget
{
        "forgotten": []
}
[root@ecs-rclone ~]# go tool pprof -text http://localhost:5572/debug/pprof/heap
Fetching profile over HTTP from http://localhost:5572/debug/pprof/heap
Saved profile in /root/pprof/pprof.rclone.alloc_objects.alloc_space.inuse_objects.inuse_space.027.pb.gz
File: rclone
Type: inuse_space
Time: Jun 9, 2023 at 3:29pm (CST)
Showing nodes accounting for 22442.12kB, 100% of 22442.12kB total
      flat  flat%   sum%        cum   cum%
 3584.87kB 15.97% 15.97%  3584.87kB 15.97%  github.com/rclone/rclone/backend/s3.s3MetadataToMap
 2560.35kB 11.41% 27.38%  6145.22kB 27.38%  github.com/rclone/rclone/backend/s3.(*Fs).Put
 2048.38kB  9.13% 36.51%  2048.38kB  9.13%  github.com/rclone/rclone/vfs.newFile
 2048.29kB  9.13% 45.64%  2048.29kB  9.13%  github.com/aws/aws-sdk-go/aws/endpoints.init
 1078.34kB  4.80% 50.44%  1078.34kB  4.80%  bazil.org/fuse/fs.(*Server).saveNode
 1024.09kB  4.56% 55.01%  1024.09kB  4.56%  strings.(*Builder).Write
 1024.06kB  4.56% 59.57%  1024.06kB  4.56%  github.com/rclone/rclone/vfs.newRWFileHandle
  768.26kB  3.42% 62.99%   768.26kB  3.42%  github.com/rclone/rclone/vfs/vfscache.(*Cache)._get
  583.01kB  2.60% 65.59%   583.01kB  2.60%  bazil.org/fuse.allocMessage
  544.67kB  2.43% 68.02%   544.67kB  2.43%  regexp/syntax.(*compiler).inst (inline)
     515kB  2.29% 70.31%      515kB  2.29%  google.golang.org/protobuf/internal/filedesc.(*File).initDecls (inline)
  514.63kB  2.29% 72.60%   514.63kB  2.29%  math/rand.NewSource (inline)
  513.50kB  2.29% 74.89%   513.50kB  2.29%  github.com/gdamore/tcell/v2/terminfo/f/foot.init.0
     513kB  2.29% 77.18%      513kB  2.29%  github.com/gdamore/encoding.init.0
  512.88kB  2.29% 79.46%   512.88kB  2.29%  sync.(*Map).dirtyLocked
  512.34kB  2.28% 81.75%   512.34kB  2.28%  github.com/colinmarc/hdfs/v2/internal/protocol/hadoop_common.init
  512.20kB  2.28% 84.03%   512.20kB  2.28%  runtime.malg
  512.07kB  2.28% 86.31%   512.07kB  2.28%  github.com/rclone/rclone/backend/s3.(*Fs).newObjectWithInfo
  512.05kB  2.28% 88.59%   512.05kB  2.28%  github.com/gabriel-vasile/mimetype.newMIME (inline)
  512.05kB  2.28% 90.87%   512.05kB  2.28%  github.com/rclone/rclone/fs/accounting.newAccountSizeName
  512.03kB  2.28% 93.16%   512.03kB  2.28%  text/template.New (inline)
  512.02kB  2.28% 95.44%   512.02kB  2.28%  math/big.nat.make (inline)
  512.02kB  2.28% 97.72%   512.02kB  2.28%  path.Join
  512.01kB  2.28%   100%   512.01kB  2.28%  net/textproto.(*Reader).ReadMIMEHeader
         0     0%   100%   583.01kB  2.60%  bazil.org/fuse.(*Conn).ReadRequest
         0     0%   100%   583.01kB  2.60%  bazil.org/fuse.getMessage (inline)
         0     0%   100%   583.01kB  2.60%  bazil.org/fuse/fs.(*Server).Serve
         0     0%   100%  5943.13kB 26.48%  bazil.org/fuse/fs.(*Server).Serve.func1
         0     0%   100%  5943.13kB 26.48%  bazil.org/fuse/fs.(*Server).handleRequest
         0     0%   100%  1078.34kB  4.80%  bazil.org/fuse/fs.(*Server).saveLookup
         0     0%   100%  5943.13kB 26.48%  bazil.org/fuse/fs.(*Server).serve
         0     0%   100%   512.88kB  2.29%  encoding/json.(*Encoder).Encode
         0     0%   100%   512.88kB  2.29%  encoding/json.(*encodeState).marshal
         0     0%   100%   512.88kB  2.29%  encoding/json.(*encodeState).reflectValue
         0     0%   100%   512.88kB  2.29%  encoding/json.typeEncoder
         0     0%   100%   512.88kB  2.29%  encoding/json.valueEncoder
         0     0%   100%  1024.09kB  4.56%  fmt.Fprintf
         0     0%   100%   512.02kB  2.28%  github.com/ProtonMail/go-crypto/brainpool.P256r1 (inline)
         0     0%   100%   512.02kB  2.28%  github.com/ProtonMail/go-crypto/brainpool.initAll
         0     0%   100%   512.02kB  2.28%  github.com/ProtonMail/go-crypto/brainpool.initP256r1
         0     0%   100%   512.02kB  2.28%  github.com/ProtonMail/go-crypto/openpgp/internal/ecc.init
         0     0%   100%   512.05kB  2.28%  github.com/gabriel-vasile/mimetype.init
         0     0%   100%   512.88kB  2.29%  github.com/go-chi/chi/v5.(*Mux).ServeHTTP
         0     0%   100%   512.88kB  2.29%  github.com/go-chi/chi/v5.(*Mux).routeHTTP
         0     0%   100%   512.88kB  2.29%  github.com/go-chi/chi/v5/middleware.SetHeader.func1.1
         0     0%   100%   512.07kB  2.28%  github.com/rclone/rclone/backend/s3.(*Fs).List
         0     0%   100%   512.07kB  2.28%  github.com/rclone/rclone/backend/s3.(*Fs).itemToDirEntry
         0     0%   100%   512.07kB  2.28%  github.com/rclone/rclone/backend/s3.(*Fs).list
         0     0%   100%   512.07kB  2.28%  github.com/rclone/rclone/backend/s3.(*Fs).listDir
         0     0%   100%   512.07kB  2.28%  github.com/rclone/rclone/backend/s3.(*Fs).listDir.func1
         0     0%   100%  3584.87kB 15.97%  github.com/rclone/rclone/backend/s3.(*Object).Update
         0     0%   100%  3584.87kB 15.97%  github.com/rclone/rclone/backend/s3.(*Object).setMetaData
         0     0%   100%   512.03kB  2.28%  github.com/rclone/rclone/cmd/gendocs.init
         0     0%   100%  4352.72kB 19.40%  github.com/rclone/rclone/cmd/mount.(*Dir).Create
         0     0%   100%   512.07kB  2.28%  github.com/rclone/rclone/cmd/mount.(*Dir).ReadDirAll
         0     0%   100%   583.01kB  2.60%  github.com/rclone/rclone/cmd/mount.mount.func2
         0     0%   100%  1024.09kB  4.56%  github.com/rclone/rclone/fs.Fingerprint
         0     0%   100%   512.05kB  2.28%  github.com/rclone/rclone/fs/accounting.(*Transfer).Account
         0     0%   100%   512.07kB  2.28%  github.com/rclone/rclone/fs/list.DirSorted
         0     0%   100%  6657.27kB 29.66%  github.com/rclone/rclone/fs/operations.Copy
         0     0%   100%   512.88kB  2.29%  github.com/rclone/rclone/fs/rc.WriteJSON
         0     0%   100%   512.88kB  2.29%  github.com/rclone/rclone/fs/rc/rcserver.(*Server).handlePost
         0     0%   100%   512.88kB  2.29%  github.com/rclone/rclone/fs/rc/rcserver.(*Server).handler
         0     0%   100%  2048.38kB  9.13%  github.com/rclone/rclone/vfs.(*Dir).Create
         0     0%   100%   512.07kB  2.28%  github.com/rclone/rclone/vfs.(*Dir).ReadDirAll
         0     0%   100%   512.07kB  2.28%  github.com/rclone/rclone/vfs.(*Dir)._readDir
         0     0%   100%  2304.34kB 10.27%  github.com/rclone/rclone/vfs.(*File).Open
         0     0%   100%   512.02kB  2.28%  github.com/rclone/rclone/vfs.(*File).Path
         0     0%   100%  1024.06kB  4.56%  github.com/rclone/rclone/vfs.(*File).openRW
         0     0%   100%   768.26kB  3.42%  github.com/rclone/rclone/vfs/vfscache.(*Cache).Exists
         0     0%   100%   768.26kB  3.42%  github.com/rclone/rclone/vfs/vfscache.(*Cache).get
         0     0%   100%  7681.36kB 34.23%  github.com/rclone/rclone/vfs/vfscache.(*Item).Close.func2
         0     0%   100%  7681.36kB 34.23%  github.com/rclone/rclone/vfs/vfscache.(*Item)._store
         0     0%   100%  1024.09kB  4.56%  github.com/rclone/rclone/vfs/vfscache.(*Item)._updateFingerprint
         0     0%   100%  7681.36kB 34.23%  github.com/rclone/rclone/vfs/vfscache.(*Item).store
         0     0%   100%  7681.36kB 34.23%  github.com/rclone/rclone/vfs/vfscache/writeback.(*WriteBack).upload
         0     0%   100%   544.67kB  2.43%  go.opencensus.io/resource.init
         0     0%   100%   514.63kB  2.29%  golang.org/x/crypto/ssh.init
         0     0%   100%      515kB  2.29%  google.golang.org/protobuf/internal/filedesc.Builder.Build
         0     0%   100%      515kB  2.29%  google.golang.org/protobuf/internal/filedesc.newRawFile
         0     0%   100%      515kB  2.29%  google.golang.org/protobuf/internal/filetype.Builder.Build
         0     0%   100%      515kB  2.29%  google.golang.org/protobuf/types/descriptorpb.file_google_protobuf_descriptor_proto_init
         0     0%   100%      515kB  2.29%  google.golang.org/protobuf/types/descriptorpb.init.0
         0     0%   100%   512.02kB  2.28%  math/big.(*Int).SetString
         0     0%   100%   512.02kB  2.28%  math/big.(*Int).scan
         0     0%   100%   512.02kB  2.28%  math/big.(*Int).setFromScanner
         0     0%   100%   512.02kB  2.28%  math/big.nat.mulAddWW
         0     0%   100%   512.02kB  2.28%  math/big.nat.scan
         0     0%   100%   512.88kB  2.29%  net/http.(*conn).serve
         0     0%   100%   512.01kB  2.28%  net/http.(*persistConn).readLoop
         0     0%   100%   512.01kB  2.28%  net/http.(*persistConn).readResponse
         0     0%   100%   512.88kB  2.29%  net/http.HandlerFunc.ServeHTTP
         0     0%   100%   512.01kB  2.28%  net/http.ReadResponse
         0     0%   100%   512.88kB  2.29%  net/http.serverHandler.ServeHTTP
         0     0%   100%   544.67kB  2.43%  regexp.Compile (inline)
         0     0%   100%   544.67kB  2.43%  regexp.MustCompile
         0     0%   100%   544.67kB  2.43%  regexp.compile
         0     0%   100%   544.67kB  2.43%  regexp/syntax.(*compiler).compile
         0     0%   100%   544.67kB  2.43%  regexp/syntax.(*compiler).quest
         0     0%   100%   544.67kB  2.43%  regexp/syntax.Compile
         0     0%   100%  6697.54kB 29.84%  runtime.doInit
         0     0%   100%  6152.87kB 27.42%  runtime.main
         0     0%   100%   512.20kB  2.28%  runtime.newproc.func1
         0     0%   100%   512.20kB  2.28%  runtime.newproc1
         0     0%   100%   512.20kB  2.28%  runtime.systemstack
         0     0%   100%   512.88kB  2.29%  sync.(*Map).LoadOrStore
         0     0%   100%   512.02kB  2.28%  sync.(*Once).Do (inline)
         0     0%   100%   512.02kB  2.28%  sync.(*Once).doSlow
         0     0%   100%   583.01kB  2.60%  sync.(*Pool).Get

It's metadata filling up the memory,
Is there any rclone command line could clear metadata from memory?

This is a tiny amount of memory - 3.5 MB.

How much memory has the machine you are running rclone on got?

Thank you for reply.

This is a server I used to test memory growth,it has 4GB memory.The actual server did not have Go debugging installed. What I mean is that when I uploaded 12507 files, the memory increased from 7.7MB to 23MB, and the actual usage environment has hundreds of thousands of files uploaded every day, increasing hundreds of MB of memory every day.

If there are 100 million files, how much memory is needed?

I uploaded ten thousands files again, and memory continues to grow.

[root@ecs-rclone ~]# go tool pprof -text http://localhost:5572/debug/pprof/heap
Fetching profile over HTTP from http://localhost:5572/debug/pprof/heap
Saved profile in /root/pprof/pprof.rclone.alloc_objects.alloc_space.inuse_objects.inuse_space.036.pb.gz
File: rclone
Type: inuse_space
Time: Jun 9, 2023 at 9:46pm (CST)
Showing nodes accounting for 49654.44kB, 100% of 49654.44kB total
      flat  flat%   sum%        cum   cum%
12803.28kB 25.78% 25.78% 12803.28kB 25.78%  github.com/rclone/rclone/backend/s3.s3MetadataToMap
 7169.75kB 14.44% 40.22%  7169.75kB 14.44%  github.com/rclone/rclone/vfs/vfscache.newItem
 4608.84kB  9.28% 49.51%  4608.84kB  9.28%  github.com/rclone/rclone/vfs.newFile
 3072.19kB  6.19% 55.69%  4608.21kB  9.28%  github.com/rclone/rclone/vfs.newRWFileHandle
 2560.09kB  5.16% 60.85%  2560.09kB  5.16%  net/textproto.(*Reader).ReadMIMEHeader
 2102.40kB  4.23% 65.08%  2102.40kB  4.23%  bazil.org/fuse/fs.(*Server).saveNode
 2048.29kB  4.13% 69.21%  2048.29kB  4.13%  github.com/aws/aws-sdk-go/aws/endpoints.init
 2048.28kB  4.13% 73.33% 14851.56kB 29.91%  github.com/rclone/rclone/backend/s3.(*Fs).Put
 2048.09kB  4.12% 77.46%  2048.09kB  4.12%  path.Join
 1536.02kB  3.09% 80.55%  1536.02kB  3.09%  github.com/rclone/rclone/vfs.(*File).addWriter
 1166.01kB  2.35% 82.90%  1166.01kB  2.35%  bazil.org/fuse.allocMessage
  768.26kB  1.55% 84.45%  7938.01kB 15.99%  github.com/rclone/rclone/vfs/vfscache.(*Cache)._get
  544.67kB  1.10% 85.54%   544.67kB  1.10%  regexp/syntax.(*compiler).inst (inline)
     515kB  1.04% 86.58%      515kB  1.04%  google.golang.org/protobuf/internal/filedesc.(*File).initDecls (inline)
  514.63kB  1.04% 87.62%   514.63kB  1.04%  math/rand.NewSource (inline)
  513.50kB  1.03% 88.65%   513.50kB  1.03%  github.com/gdamore/tcell/v2/terminfo/f/foot.init.0
     513kB  1.03% 89.68%      513kB  1.03%  github.com/gdamore/encoding.init.0
  512.88kB  1.03% 90.72%   512.88kB  1.03%  sync.(*Map).dirtyLocked
  512.56kB  1.03% 91.75%   512.56kB  1.03%  github.com/rclone/rclone/backend/local.newFadvise
  512.34kB  1.03% 92.78%   512.34kB  1.03%  github.com/colinmarc/hdfs/v2/internal/protocol/hadoop_common.init
  512.20kB  1.03% 93.81%   512.20kB  1.03%  runtime.malg
  512.05kB  1.03% 94.84%   512.05kB  1.03%  github.com/gabriel-vasile/mimetype.newMIME (inline)
  512.03kB  1.03% 95.88%   512.03kB  1.03%  text/template.New (inline)
  512.02kB  1.03% 96.91%   512.02kB  1.03%  github.com/rclone/rclone/vfs.newDir
  512.02kB  1.03% 97.94%   512.02kB  1.03%  math/big.nat.make (inline)
  512.01kB  1.03% 98.97%  1678.02kB  3.38%  bazil.org/fuse.(*Conn).ReadRequest
  512.01kB  1.03%   100% 18690.98kB 37.64%  github.com/rclone/rclone/cmd/mount.(*Dir).Create
         0     0%   100%  1166.01kB  2.35%  bazil.org/fuse.getMessage (inline)
         0     0%   100%  1678.02kB  3.38%  bazil.org/fuse/fs.(*Server).Serve
         0     0%   100% 22329.59kB 44.97%  bazil.org/fuse/fs.(*Server).Serve.func1
         0     0%   100% 22329.59kB 44.97%  bazil.org/fuse/fs.(*Server).handleRequest
         0     0%   100%  2102.40kB  4.23%  bazil.org/fuse/fs.(*Server).saveLookup
         0     0%   100% 22329.59kB 44.97%  bazil.org/fuse/fs.(*Server).serve
         0     0%   100%   512.88kB  1.03%  encoding/json.(*Encoder).Encode
         0     0%   100%   512.88kB  1.03%  encoding/json.(*encodeState).marshal
         0     0%   100%   512.88kB  1.03%  encoding/json.(*encodeState).reflectValue
         0     0%   100%   512.88kB  1.03%  encoding/json.typeEncoder
         0     0%   100%   512.88kB  1.03%  encoding/json.valueEncoder
         0     0%   100%   512.02kB  1.03%  github.com/ProtonMail/go-crypto/brainpool.P256r1 (inline)
         0     0%   100%   512.02kB  1.03%  github.com/ProtonMail/go-crypto/brainpool.initAll
         0     0%   100%   512.02kB  1.03%  github.com/ProtonMail/go-crypto/brainpool.initP256r1
         0     0%   100%   512.02kB  1.03%  github.com/ProtonMail/go-crypto/openpgp/internal/ecc.init
         0     0%   100%   512.05kB  1.03%  github.com/gabriel-vasile/mimetype.init
         0     0%   100%   512.88kB  1.03%  github.com/go-chi/chi/v5.(*Mux).ServeHTTP
         0     0%   100%   512.88kB  1.03%  github.com/go-chi/chi/v5.(*Mux).routeHTTP
         0     0%   100%   512.88kB  1.03%  github.com/go-chi/chi/v5/middleware.SetHeader.func1.1
         0     0%   100%   512.56kB  1.03%  github.com/rclone/rclone/backend/local.(*Object).Open
         0     0%   100%   512.56kB  1.03%  github.com/rclone/rclone/backend/local.newFadviseReadCloser
         0     0%   100% 12803.28kB 25.78%  github.com/rclone/rclone/backend/s3.(*Object).Update
         0     0%   100% 12803.28kB 25.78%  github.com/rclone/rclone/backend/s3.(*Object).setMetaData
         0     0%   100%   512.03kB  1.03%  github.com/rclone/rclone/cmd/gendocs.init
         0     0%   100%   512.02kB  1.03%  github.com/rclone/rclone/cmd/mount.(*Dir).Mkdir
         0     0%   100%  1024.19kB  2.06%  github.com/rclone/rclone/cmd/mount.(*Dir).ReadDirAll
         0     0%   100%  1678.02kB  3.38%  github.com/rclone/rclone/cmd/mount.mount.func2
         0     0%   100%   512.56kB  1.03%  github.com/rclone/rclone/fs/operations.(*ReOpen).open
         0     0%   100% 15364.12kB 30.94%  github.com/rclone/rclone/fs/operations.Copy
         0     0%   100%   512.56kB  1.03%  github.com/rclone/rclone/fs/operations.NewReOpen
         0     0%   100%   512.88kB  1.03%  github.com/rclone/rclone/fs/rc.WriteJSON
         0     0%   100%   512.88kB  1.03%  github.com/rclone/rclone/fs/rc/rcserver.(*Server).handlePost
         0     0%   100%   512.88kB  1.03%  github.com/rclone/rclone/fs/rc/rcserver.(*Server).handler
         0     0%   100%  3584.66kB  7.22%  github.com/rclone/rclone/vfs.(*Dir).Create
         0     0%   100%   512.02kB  1.03%  github.com/rclone/rclone/vfs.(*Dir).Mkdir
         0     0%   100%  1024.19kB  2.06%  github.com/rclone/rclone/vfs.(*Dir).ReadDirAll
         0     0%   100%  1024.19kB  2.06%  github.com/rclone/rclone/vfs.(*Dir)._readDir
         0     0%   100%  1024.19kB  2.06%  github.com/rclone/rclone/vfs.(*Dir)._readDirFromEntries
         0     0%   100% 14594.31kB 29.39%  github.com/rclone/rclone/vfs.(*File).Open
         0     0%   100%  2048.09kB  4.12%  github.com/rclone/rclone/vfs.(*File).Path
         0     0%   100%  4608.21kB  9.28%  github.com/rclone/rclone/vfs.(*File).openRW
         0     0%   100%  7938.01kB 15.99%  github.com/rclone/rclone/vfs/vfscache.(*Cache).Exists
         0     0%   100%  7938.01kB 15.99%  github.com/rclone/rclone/vfs/vfscache.(*Cache).get
         0     0%   100% 15364.12kB 30.94%  github.com/rclone/rclone/vfs/vfscache.(*Item).Close.func2
         0     0%   100% 15364.12kB 30.94%  github.com/rclone/rclone/vfs/vfscache.(*Item)._store
         0     0%   100% 15364.12kB 30.94%  github.com/rclone/rclone/vfs/vfscache.(*Item).store
         0     0%   100% 15364.12kB 30.94%  github.com/rclone/rclone/vfs/vfscache/writeback.(*WriteBack).upload
         0     0%   100%   544.67kB  1.10%  go.opencensus.io/resource.init
         0     0%   100%   514.63kB  1.04%  golang.org/x/crypto/ssh.init
         0     0%   100%      515kB  1.04%  google.golang.org/protobuf/internal/filedesc.Builder.Build
         0     0%   100%      515kB  1.04%  google.golang.org/protobuf/internal/filedesc.newRawFile
         0     0%   100%      515kB  1.04%  google.golang.org/protobuf/internal/filetype.Builder.Build
         0     0%   100%      515kB  1.04%  google.golang.org/protobuf/types/descriptorpb.file_google_protobuf_descriptor_proto_init
         0     0%   100%      515kB  1.04%  google.golang.org/protobuf/types/descriptorpb.init.0
         0     0%   100%   512.02kB  1.03%  math/big.(*Int).SetString
         0     0%   100%   512.02kB  1.03%  math/big.(*Int).scan
         0     0%   100%   512.02kB  1.03%  math/big.(*Int).setFromScanner
         0     0%   100%   512.02kB  1.03%  math/big.nat.mulAddWW
         0     0%   100%   512.02kB  1.03%  math/big.nat.scan
         0     0%   100%   512.88kB  1.03%  net/http.(*conn).serve
         0     0%   100%  2560.09kB  5.16%  net/http.(*persistConn).readLoop
         0     0%   100%  2560.09kB  5.16%  net/http.(*persistConn).readResponse
         0     0%   100%   512.88kB  1.03%  net/http.HandlerFunc.ServeHTTP
         0     0%   100%  2560.09kB  5.16%  net/http.ReadResponse
         0     0%   100%   512.88kB  1.03%  net/http.serverHandler.ServeHTTP
         0     0%   100%   544.67kB  1.10%  regexp.Compile (inline)
         0     0%   100%   544.67kB  1.10%  regexp.MustCompile
         0     0%   100%   544.67kB  1.10%  regexp.compile
         0     0%   100%   544.67kB  1.10%  regexp/syntax.(*compiler).compile
         0     0%   100%   544.67kB  1.10%  regexp/syntax.(*compiler).quest
         0     0%   100%   544.67kB  1.10%  regexp/syntax.Compile
         0     0%   100%  6697.54kB 13.49%  runtime.doInit
         0     0%   100%  6152.87kB 12.39%  runtime.main
         0     0%   100%   512.20kB  1.03%  runtime.newproc.func1
         0     0%   100%   512.20kB  1.03%  runtime.newproc1
         0     0%   100%   512.20kB  1.03%  runtime.systemstack
         0     0%   100%   512.88kB  1.03%  sync.(*Map).LoadOrStore
         0     0%   100%   512.02kB  1.03%  sync.(*Once).Do (inline)
         0     0%   100%   512.02kB  1.03%  sync.(*Once).doSlow
         0     0%   100%  1166.01kB  2.35%  sync.(*Pool).Get

So that is about (23-7.7)/12507 1.2 kB per file which sounds about right for files stored in memory.

I would say these are being stored in the directory cache. It looks like from your experiments that vfs/forget is not clearing the directory cache of these though.

Thank you for your reply.

What is the difference between dir-cache-time and rclone rc vfs/forget? When dir-cache-time expires, the directory cache seems release,but the memory usage didn't decrease. Rclone rc vfs/forget can release most memory.

When dir-cache-time expired,memory seems not change, always 246.28MB used。But accessing the vfs directory takes longer time,which makes me feel rclone has released directories and files cache.

[root@ecs-rclone /]# go tool pprof -text http://localhost:5572/debug/pprof/heap
Fetching profile over HTTP from http://localhost:5572/debug/pprof/heap
Saved profile in /root/pprof/pprof.rclone.alloc_objects.alloc_space.inuse_objects.inuse_space.147.pb.gz
File: rclone
Type: inuse_space
Time: Jun 21, 2023 at 10:44am (CST)
Showing nodes accounting for 242.75MB, 98.56% of 246.28MB total
Dropped 45 nodes (cum <= 1.23MB)
      flat  flat%   sum%        cum   cum%
   69.51MB 28.22% 28.22%    69.51MB 28.22%  github.com/rclone/rclone/vfs.newDir
   28.51MB 11.57% 39.80%    28.51MB 11.57%  github.com/rclone/rclone/vfs.newFile
   27.44MB 11.14% 50.94%    27.44MB 11.14%  bazil.org/fuse/fs.(*Server).saveNode
   26.79MB 10.88% 61.82%   124.80MB 50.67%  github.com/rclone/rclone/vfs.(*Dir)._readDirFromEntries
   26.50MB 10.76% 72.58%       30MB 12.18%  github.com/rclone/rclone/backend/s3.(*Fs).newObjectWithInfo
   26.50MB 10.76% 83.34%    30.50MB 12.38%  github.com/aws/aws-sdk-go/private/protocol/xml/xmlutil.XMLToStruct
   24.50MB  9.95% 93.29%    24.50MB  9.95%  github.com/rclone/rclone/fs.NewDir (inline)
       4MB  1.62% 94.91%        4MB  1.62%  encoding/xml.CharData.Copy (inline)
    3.50MB  1.42% 96.33%     3.50MB  1.42%  github.com/rclone/rclone/backend/s3.stringClonePointer (inline)
       3MB  1.22% 97.55%    88.04MB 35.75%  github.com/rclone/rclone/cmd/mount.(*Dir).Lookup
    2.50MB  1.02% 98.56%     2.50MB  1.02%  github.com/aws/aws-sdk-go/aws/endpoints.init
         0     0% 98.56%   240.25MB 97.55%  bazil.org/fuse/fs.(*Server).Serve.func1
         0     0% 98.56%   240.25MB 97.55%  bazil.org/fuse/fs.(*Server).handleRequest
         0     0% 98.56%    27.44MB 11.14%  bazil.org/fuse/fs.(*Server).saveLookup
         0     0% 98.56%   240.25MB 97.55%  bazil.org/fuse/fs.(*Server).serve
         0     0% 98.56%    30.50MB 12.38%  github.com/aws/aws-sdk-go/aws/request.(*HandlerList).Run
         0     0% 98.56%    30.50MB 12.38%  github.com/aws/aws-sdk-go/aws/request.(*Request).Send
         0     0% 98.56%    30.50MB 12.38%  github.com/aws/aws-sdk-go/aws/request.(*Request).sendRequest
         0     0% 98.56%    30.50MB 12.38%  github.com/aws/aws-sdk-go/private/protocol/restxml.Unmarshal
         0     0% 98.56%    30.50MB 12.38%  github.com/aws/aws-sdk-go/private/protocol/xml/xmlutil.UnmarshalXML
         0     0% 98.56%    30.50MB 12.38%  github.com/aws/aws-sdk-go/service/s3.(*S3).ListObjectsWithContext
         0     0% 98.56%    85.01MB 34.52%  github.com/rclone/rclone/backend/s3.(*Fs).List
         0     0% 98.56%    54.51MB 22.13%  github.com/rclone/rclone/backend/s3.(*Fs).itemToDirEntry
         0     0% 98.56%    85.01MB 34.52%  github.com/rclone/rclone/backend/s3.(*Fs).list
         0     0% 98.56%    30.50MB 12.38%  github.com/rclone/rclone/backend/s3.(*Fs).list.func1
         0     0% 98.56%    85.01MB 34.52%  github.com/rclone/rclone/backend/s3.(*Fs).listDir
         0     0% 98.56%    54.51MB 22.13%  github.com/rclone/rclone/backend/s3.(*Fs).listDir.func1
         0     0% 98.56%    30.50MB 12.38%  github.com/rclone/rclone/backend/s3.(*v1List).List
         0     0% 98.56%   124.77MB 50.66%  github.com/rclone/rclone/cmd/mount.(*Dir).ReadDirAll
         0     0% 98.56%    30.50MB 12.38%  github.com/rclone/rclone/fs.pacerInvoker
         0     0% 98.56%    85.01MB 34.52%  github.com/rclone/rclone/fs/list.DirSorted
         0     0% 98.56%    30.50MB 12.38%  github.com/rclone/rclone/lib/pacer.(*Pacer).Call
         0     0% 98.56%    30.50MB 12.38%  github.com/rclone/rclone/lib/pacer.(*Pacer).call
         0     0% 98.56%   124.77MB 50.66%  github.com/rclone/rclone/vfs.(*Dir).ReadDirAll
         0     0% 98.56%    85.04MB 34.53%  github.com/rclone/rclone/vfs.(*Dir).Stat
         0     0% 98.56%   209.81MB 85.19%  github.com/rclone/rclone/vfs.(*Dir)._readDir
         0     0% 98.56%    85.04MB 34.53%  github.com/rclone/rclone/vfs.(*Dir).stat
         0     0% 98.56%     5.54MB  2.25%  runtime.doInit
         0     0% 98.56%     5.54MB  2.25%  runtime.main

After using rclone rc vfs/forget, memory release 100MB.This cmd will take a while, that is why I felt it didn't take effect before

[root@ecs-rclone /]# go tool pprof -text http://localhost:5572/debug/pprof/heap
Fetching profile over HTTP from http://localhost:5572/debug/pprof/heap
Saved profile in /root/pprof/pprof.rclone.alloc_objects.alloc_space.inuse_objects.inuse_space.158.pb.gz
File: rclone
Type: inuse_space
Time: Jun 21, 2023 at 11:10am (CST)
Showing nodes accounting for 142.95MB, 97.54% of 146.56MB total
Dropped 50 nodes (cum <= 0.73MB)
      flat  flat%   sum%        cum   cum%
   28.01MB 19.11% 19.11%    28.01MB 19.11%  github.com/rclone/rclone/vfs.newFile
   27.44MB 18.72% 37.83%    27.44MB 18.72%  bazil.org/fuse/fs.(*Server).saveNode
   26.50MB 18.08% 55.92%       30MB 20.47%  github.com/rclone/rclone/backend/s3.(*Fs).newObjectWithInfo
      17MB 11.60% 67.52%       21MB 14.33%  github.com/aws/aws-sdk-go/private/protocol/xml/xmlutil.XMLToStruct
      15MB 10.24% 77.75%       15MB 10.24%  github.com/rclone/rclone/vfs.newDir
       9MB  6.14% 83.89%        9MB  6.14%  github.com/rclone/rclone/fs.NewDir (inline)
    6.50MB  4.44% 88.33%     6.50MB  4.44%  github.com/rclone/rclone/vfs.(*Dir).ForgetAll
       4MB  2.73% 91.06%        4MB  2.73%  encoding/xml.CharData.Copy (inline)
    3.50MB  2.39% 93.45%     3.50MB  2.39%  github.com/rclone/rclone/backend/s3.stringClonePointer (inline)
       3MB  2.05% 95.49%        3MB  2.05%  github.com/rclone/rclone/cmd/mount.(*Dir).Lookup
    2.50MB  1.71% 97.20%     2.50MB  1.71%  github.com/aws/aws-sdk-go/aws/endpoints.init
    0.50MB  0.34% 97.54%     1.03MB   0.7%  mime.setExtensionType
         0     0% 97.54%   133.45MB 91.06%  bazil.org/fuse/fs.(*Server).Serve.func1
         0     0% 97.54%   133.45MB 91.06%  bazil.org/fuse/fs.(*Server).handleRequest
         0     0% 97.54%    27.44MB 18.72%  bazil.org/fuse/fs.(*Server).saveLookup
         0     0% 97.54%   133.45MB 91.06%  bazil.org/fuse/fs.(*Server).serve
         0     0% 97.54%       21MB 14.33%  github.com/aws/aws-sdk-go/aws/request.(*HandlerList).Run
         0     0% 97.54%       21MB 14.33%  github.com/aws/aws-sdk-go/aws/request.(*Request).Send
         0     0% 97.54%       21MB 14.33%  github.com/aws/aws-sdk-go/aws/request.(*Request).sendRequest
         0     0% 97.54%       21MB 14.33%  github.com/aws/aws-sdk-go/private/protocol/restxml.Unmarshal
         0     0% 97.54%       21MB 14.33%  github.com/aws/aws-sdk-go/private/protocol/xml/xmlutil.UnmarshalXML
         0     0% 97.54%       21MB 14.33%  github.com/aws/aws-sdk-go/service/s3.(*S3).ListObjectsWithContext
         0     0% 97.54%     6.50MB  4.44%  github.com/go-chi/chi/v5.(*Mux).ServeHTTP
         0     0% 97.54%     6.50MB  4.44%  github.com/go-chi/chi/v5.(*Mux).routeHTTP
         0     0% 97.54%     6.50MB  4.44%  github.com/go-chi/chi/v5/middleware.SetHeader.func1.1
         0     0% 97.54%    60.01MB 40.94%  github.com/rclone/rclone/backend/s3.(*Fs).List
         0     0% 97.54%       39MB 26.61%  github.com/rclone/rclone/backend/s3.(*Fs).itemToDirEntry
         0     0% 97.54%    60.01MB 40.94%  github.com/rclone/rclone/backend/s3.(*Fs).list
         0     0% 97.54%       21MB 14.33%  github.com/rclone/rclone/backend/s3.(*Fs).list.func1
         0     0% 97.54%    60.01MB 40.94%  github.com/rclone/rclone/backend/s3.(*Fs).listDir
         0     0% 97.54%       39MB 26.61%  github.com/rclone/rclone/backend/s3.(*Fs).listDir.func1
         0     0% 97.54%       21MB 14.33%  github.com/rclone/rclone/backend/s3.(*v1List).List
         0     0% 97.54%   103.01MB 70.29%  github.com/rclone/rclone/cmd/mount.(*Dir).ReadDirAll
         0     0% 97.54%       21MB 14.33%  github.com/rclone/rclone/fs.pacerInvoker
         0     0% 97.54%    60.01MB 40.94%  github.com/rclone/rclone/fs/list.DirSorted
         0     0% 97.54%     6.50MB  4.44%  github.com/rclone/rclone/fs/rc/jobs.(*Job).run
         0     0% 97.54%     6.50MB  4.44%  github.com/rclone/rclone/fs/rc/jobs.(*Jobs).NewJob
         0     0% 97.54%     6.50MB  4.44%  github.com/rclone/rclone/fs/rc/jobs.NewJob (inline)
         0     0% 97.54%     6.50MB  4.44%  github.com/rclone/rclone/fs/rc/rcserver.(*Server).handlePost
         0     0% 97.54%     6.50MB  4.44%  github.com/rclone/rclone/fs/rc/rcserver.(*Server).handler
         0     0% 97.54%       21MB 14.33%  github.com/rclone/rclone/lib/pacer.(*Pacer).Call
         0     0% 97.54%       21MB 14.33%  github.com/rclone/rclone/lib/pacer.(*Pacer).call
         0     0% 97.54%   103.01MB 70.29%  github.com/rclone/rclone/vfs.(*Dir).ReadDirAll
         0     0% 97.54%   103.01MB 70.29%  github.com/rclone/rclone/vfs.(*Dir)._readDir
         0     0% 97.54%    43.01MB 29.35%  github.com/rclone/rclone/vfs.(*Dir)._readDirFromEntries
         0     0% 97.54%     6.50MB  4.44%  github.com/rclone/rclone/vfs.rcForget
         0     0% 97.54%     6.50MB  4.44%  net/http.(*conn).serve
         0     0% 97.54%     6.50MB  4.44%  net/http.HandlerFunc.ServeHTTP
         0     0% 97.54%     6.50MB  4.44%  net/http.serverHandler.ServeHTTP
         0     0% 97.54%     5.54MB  3.78%  runtime.doInit
         0     0% 97.54%     5.54MB  3.78%  runtime.main

This is something we are trying to fix here:

So you could try that branch or wait for it to be merged.

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.