Probable memory leak
Version
rclone v1.57.0-DEV
- os/version: gentoo 2.8 (64 bit)
- os/kernel: 5.14.14-gentoo-x86_64 (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.17.7
- go/linking: dynamic
- go/tags: none
Yes I've build it my self to add pprof but with the official release it just takes more and more memory until it gets killed by the OMM
storage system
Blomp (Swift)
The command
~/rclone-fixed -vv --transfers=8 --cache-dir=/storage/hot/rclone-cache/ --vfs-cache-mode full --config ~/.config/rclone/blomp.conf mount blomp-arch-hash:/final-backup/ remote
There is no difference if it's copy or sync in all cases it will use up all memory and get killed by the omm
The rclone config
# cat ~/.config/rclone/blomp.conf
[blomp-remote]
type = swift
user = CENSORED
key = CENSORED
auth = CENSORED
tenant = storage
auth_version = 2
leave_parts_on_error = true
[blomp-alias]
type = alias
remote = blomp-remote:CENSORED
[blomp-arch]
type = compress
remote = blomp-alias:arch
level = 9
ram_cache_limit = 130Mi
[blomp-arch-hash]
type = hasher
remote = blomp-arch:
hashes = md5,sha1,sha256
The interesting part of ppref
heap profile: 4290: 5142851824 [113564: 644570274512] @ heap/1048576
3379: 3764584448 [3379: 3764584448] @ 0x1402416 0xd01d32 0x1403607 0xda37e7 0xd27f0e 0x14381d0 0x14381af 0x1437fd8 0x143b1f0 0x16a0b4d 0x1361315 0x1db2a1e 0x1db2d8c 0x1db38f4 0x1da51b8 0xcf76c1
# 0x1402415 github.com/buengese/sgzip.NewWriterLevel.func1+0x35 [REDACTED]/go/pkg/mod/github.com/buengese/sgzip@v0.1.1/gzip.go:91
# 0xd01d31 sync.(*Pool).Get+0xb1 /usr/lib/go/src/sync/pool.go:148
# 0x1403606 github.com/buengese/sgzip.(*Writer).Write+0x2e6 [REDACTED]/go/pkg/mod/github.com/buengese/sgzip@v0.1.1/gzip.go:390
# 0xda37e6 bytes.(*Reader).WriteTo+0x86 /usr/lib/go/src/bytes/reader.go:144
# 0xd27f0d io.copyBuffer+0x16d /usr/lib/go/src/io/io.go:405
# 0x14381cf io.Copy+0x6f /usr/lib/go/src/io/io.go:382
# 0x14381ae github.com/rclone/rclone/backend/compress.isCompressible+0x4e [REDACTED]/[REDACTED]/rclone/backend/compress/compress.go:400
# 0x1437fd7 github.com/rclone/rclone/backend/compress.checkCompressAndType+0x177 [REDACTED]/[REDACTED]/rclone/backend/compress/compress.go:384
# 0x143b1ef github.com/rclone/rclone/backend/compress.(*Fs).Put+0x1af [REDACTED]/[REDACTED]/rclone/backend/compress/compress.go:683
# 0x16a0b4c github.com/rclone/rclone/backend/hasher.(*Fs).Put+0x36c [REDACTED]/[REDACTED]/rclone/backend/hasher/object.go:212
# 0x1361314 github.com/rclone/rclone/fs/operations.Copy+0x1a94 [REDACTED]/[REDACTED]/rclone/fs/operations/operations.go:472
# 0x1db2a1d github.com/rclone/rclone/vfs/vfscache.(*Item)._store+0x1bd [REDACTED]/[REDACTED]/rclone/vfs/vfscache/item.go:584
# 0x1db2d8b github.com/rclone/rclone/vfs/vfscache.(*Item).store+0xcb [REDACTED]/[REDACTED]/rclone/vfs/vfscache/item.go:615
# 0x1db38f3 github.com/rclone/rclone/vfs/vfscache.(*Item).Close.func2+0x33 [REDACTED]/[REDACTED]/rclone/vfs/vfscache/item.go:714
# 0x1da51b7 github.com/rclone/rclone/vfs/vfscache/writeback.(*WriteBack).upload+0x137 [REDACTED]/[REDACTED]/rclone/vfs/vfscache/writeback/writeback.go:354
5: 681574400 [3877: 528492789760] @ 0x14385dd 0x1439805 0x143ad4d 0x143b2fd 0x16a0b4d 0x1361315 0x1db2a1e 0x1db2d8c 0x1db38f4 0x1da51b8 0xcf76c1
# 0x14385dc github.com/rclone/rclone/backend/compress.(*Fs).rcat+0x9c [REDACTED]/[REDACTED]/rclone/backend/compress/compress.go:438
# 0x1439804 github.com/rclone/rclone/backend/compress.(*Fs).putCompress+0x804 [REDACTED]/[REDACTED]/rclone/backend/compress/compress.go:535
# 0x143ad4c github.com/rclone/rclone/backend/compress.(*Fs).putWithCustomFunctions+0xac [REDACTED]/[REDACTED]/rclone/backend/compress/compress.go:650
# 0x143b2fc github.com/rclone/rclone/backend/compress.(*Fs).Put+0x2bc [REDACTED]/[REDACTED]/rclone/backend/compress/compress.go:687
# 0x16a0b4c github.com/rclone/rclone/backend/hasher.(*Fs).Put+0x36c [REDACTED]/[REDACTED]/rclone/backend/hasher/object.go:212
# 0x1361314 github.com/rclone/rclone/fs/operations.Copy+0x1a94 [REDACTED]/[REDACTED]/rclone/fs/operations/operations.go:472
# 0x1db2a1d github.com/rclone/rclone/vfs/vfscache.(*Item)._store+0x1bd [REDACTED]/[REDACTED]/rclone/vfs/vfscache/item.go:584
# 0x1db2d8b github.com/rclone/rclone/vfs/vfscache.(*Item).store+0xcb [REDACTED]/[REDACTED]/rclone/vfs/vfscache/item.go:615
# 0x1db38f3 github.com/rclone/rclone/vfs/vfscache.(*Item).Close.func2+0x33 [REDACTED]/[REDACTED]/rclone/vfs/vfscache/item.go:714
# 0x1da51b7 github.com/rclone/rclone/vfs/vfscache/writeback.(*WriteBack).upload+0x137 [REDACTED]/[REDACTED]/rclone/vfs/vfscache/writeback/writeback.go:354
the rest of ppref heap info rclone memory leak - Pastebin Service