cberni
July 22, 2022, 10:01pm
1
rclone increases memory linearly during usage.--use-mmap. Any way to help clean the memory usage?
bellow the memory usage at 10:33am and than at 6:30pm...
Fetching profile over HTTP from http://localhost:5572/debug/pprof/heap
Saved profile in /root/pprof/pprof.rclone.alloc_objects.alloc_space.inuse_objects.inuse_space.002.pb.gz
File: rclone
Type: inuse_space
Time: Jul 22, 2022 at 10:33am (-03)
Showing nodes accounting for 1446.29MB, 97.99% of 1475.99MB total
Dropped 153 nodes (cum <= 7.38MB)
flat flat% sum% cum cum%
464MB 31.44% 31.44% 464MB 31.44% google.golang.org/api/internal/gensupport.NewMediaBuffer (inline)
157.02MB 10.64% 42.07% 157.02MB 10.64% encoding/json.(*decodeState).literalStore
118.02MB 8.00% 50.07% 118.02MB 8.00% github.com/rclone/rclone/vfs.newFile
75.02MB 5.08% 55.15% 96.02MB 6.51% github.com/rclone/rclone/vfs/vfscache.newItem
64.51MB 4.37% 59.52% 74.01MB 5.01% github.com/rclone/rclone/fs/hash.(*MultiHasher).Sums
59.51MB 4.03% 63.56% 106.51MB 7.22% github.com/rclone/rclone/backend/local.(*Fs).newObject
56.01MB 3.79% 67.35% 62.01MB 4.20% github.com/rclone/rclone/fs/hash.StreamTypes
47.50MB 3.22% 70.57% 47.50MB 3.22% path.Join
47MB 3.18% 73.75% 47MB 3.18% strings.(*Builder).grow (inline)
39.50MB 2.68% 76.43% 52.04MB 3.53% github.com/rclone/rclone/backend/googlecloudstorage.(*Fs).newObjectWithInfo
36.50MB 2.47% 78.90% 36.50MB 2.47% encoding/hex.EncodeToString (inline)
34.99MB 2.37% 81.27% 34.99MB 2.37% github.com/rclone/rclone/vfs.(*Dir).addObject
31.50MB 2.13% 83.41% 31.50MB 2.13% github.com/rclone/rclone/backend/union/upstream.(*Fs).WrapObject (inline)
30.79MB 2.09% 85.49% 110.80MB 7.51% github.com/rclone/rclone/vfs.(*Dir)._readDirFromEntries
28MB 1.90% 87.39% 545.53MB 36.96% github.com/rclone/rclone/backend/googlecloudstorage.(*Fs).Put
27.95MB 1.89% 89.28% 27.95MB 1.89% bazil.org/fuse/fs.(*Server).saveNode
27.50MB 1.86% 91.15% 27.50MB 1.86% github.com/rclone/rclone/backend/union.(*Fs).wrapEntries
21MB 1.42% 92.57% 21MB 1.42% sync.NewCond (inline)
20MB 1.36% 93.93% 59.98MB 4.06% github.com/rclone/rclone/vfs.newRWFileHandle
15.50MB 1.05% 94.98% 15.50MB 1.05% github.com/rclone/rclone/vfs.newDir
14.50MB 0.98% 95.96% 32MB 2.17% github.com/rclone/rclone/backend/union.(*Fs).mergeDirEntries
9MB 0.61% 96.57% 19MB 1.29% github.com/rclone/rclone/backend/union.(*Fs).put
7.50MB 0.51% 97.08% 7.50MB 0.51% github.com/rclone/rclone/fs.NewDir (inline)
7MB 0.47% 97.55% 149.80MB 10.15% github.com/rclone/rclone/cmd/mount.(*Dir).Lookup
3.46MB 0.23% 97.78% 99.47MB 6.74% github.com/rclone/rclone/vfs/vfscache.(*Cache)._get
3MB 0.2% 97.99% 199MB 13.48% github.com/rclone/rclone/cmd/mount.(*Dir).Create
0 0% 97.99% 480.45MB 32.55% bazil.org/fuse/fs.(*Server).Serve.func1
0 0% 97.99% 480.45MB 32.55% bazil.org/fuse/fs.(*Server).handleRequest
0 0% 97.99% 27.95MB 1.89% bazil.org/fuse/fs.(*Server).saveLookup
0 0% 97.99% 480.45MB 32.55% bazil.org/fuse/fs.(*Server).serve
0 0% 97.99% 157.54MB 10.67% encoding/json.(*Decoder).Decode
0 0% 97.99% 108.01MB 7.32% encoding/json.(*decodeState).array
0 0% 97.99% 157.02MB 10.64% encoding/json.(*decodeState).object
0 0% 97.99% 157.02MB 10.64% encoding/json.(*decodeState).unmarshal
0 0% 97.99% 157.02MB 10.64% encoding/json.(*decodeState).value
0 0% 97.99% 165.53MB 11.22% github.com/rclone/rclone/backend/googlecloudstorage.(*Fs).List
0 0% 97.99% 57.01MB 3.86% github.com/rclone/rclone/backend/googlecloudstorage.(*Fs).itemToDirEntry
0 0% 97.99% 165.53MB 11.22% github.com/rclone/rclone/backend/googlecloudstorage.(*Fs).list
0 0% 97.99% 108.53MB 7.35% github.com/rclone/rclone/backend/googlecloudstorage.(*Fs).list.func1
0 0% 97.99% 165.53MB 11.22% github.com/rclone/rclone/backend/googlecloudstorage.(*Fs).listDir
0 0% 97.99% 57.01MB 3.86% github.com/rclone/rclone/backend/googlecloudstorage.(*Fs).listDir.func1
0 0% 97.99% 523.52MB 35.47% github.com/rclone/rclone/backend/googlecloudstorage.(*Object).Update
0 0% 97.99% 513.52MB 34.79% github.com/rclone/rclone/backend/googlecloudstorage.(*Object).Update.func1
0 0% 97.99% 21MB 1.42% github.com/rclone/rclone/backend/googlecloudstorage.(*Object).setMetaData
0 0% 97.99% 83.01MB 5.62% github.com/rclone/rclone/backend/local.(*Fs).List
0 0% 97.99% 118.05MB 8.00% github.com/rclone/rclone/backend/local.(*Fs).Put
0 0% 97.99% 26.50MB 1.80% github.com/rclone/rclone/backend/local.(*Fs).cleanRemote
0 0% 97.99% 47MB 3.18% github.com/rclone/rclone/backend/local.(*Fs).localPath
0 0% 97.99% 55.50MB 3.76% github.com/rclone/rclone/backend/local.(*Fs).newObjectWithInfo
0 0% 97.99% 62.01MB 4.20% github.com/rclone/rclone/backend/local.(*Object).Hash
0 0% 97.99% 75.04MB 5.08% github.com/rclone/rclone/backend/local.(*Object).Update
0 0% 97.99% 32MB 2.17% github.com/rclone/rclone/backend/union.(*Fs).List
0 0% 97.99% 267.04MB 18.09% github.com/rclone/rclone/backend/union.(*Fs).List.func1
0 0% 97.99% 19MB 1.29% github.com/rclone/rclone/backend/union.(*Fs).Put
0 0% 97.99% 677.57MB 45.91% github.com/rclone/rclone/backend/union.(*Fs).put.func1
0 0% 97.99% 14MB 0.95% github.com/rclone/rclone/backend/union.(*Object).Update.func1
0 0% 97.99% 960.15MB 65.05% github.com/rclone/rclone/backend/union.multithread.func1
0 0% 97.99% 663.57MB 44.96% github.com/rclone/rclone/backend/union/upstream.(*Fs).Put
0 0% 97.99% 18.50MB 1.25% github.com/rclone/rclone/backend/union/upstream.(*Fs).WrapEntry
0 0% 97.99% 14MB 0.95% github.com/rclone/rclone/backend/union/upstream.(*Object).Update
0 0% 97.99% 36.96MB 2.50% github.com/rclone/rclone/cmd/mount.(*File).Open
0 0% 97.99% 62.01MB 4.20% github.com/rclone/rclone/cmd/mount.(*FileHandle).Flush
0 0% 97.99% 623.58MB 42.25% github.com/rclone/rclone/fs.pacerInvoker
0 0% 97.99% 32MB 2.17% github.com/rclone/rclone/fs/list.DirSorted
0 0% 97.99% 19MB 1.29% github.com/rclone/rclone/fs/operations.Copy
0 0% 97.99% 110.06MB 7.46% github.com/rclone/rclone/lib/pacer.(*Pacer).Call
0 0% 97.99% 513.52MB 34.79% github.com/rclone/rclone/lib/pacer.(*Pacer).CallNoRetry (inline)
0 0% 97.99% 623.58MB 42.25% github.com/rclone/rclone/lib/pacer.(*Pacer).call
0 0% 97.99% 53.01MB 3.59% github.com/rclone/rclone/vfs.(*Dir).Create
0 0% 97.99% 142.80MB 9.68% github.com/rclone/rclone/vfs.(*Dir).Stat
0 0% 97.99% 142.80MB 9.68% github.com/rclone/rclone/vfs.(*Dir)._readDir
0 0% 97.99% 142.80MB 9.68% github.com/rclone/rclone/vfs.(*Dir).stat
0 0% 97.99% 179.95MB 12.19% github.com/rclone/rclone/vfs.(*File).Open
0 0% 97.99% 20.50MB 1.39% github.com/rclone/rclone/vfs.(*File).Path
0 0% 97.99% 59.98MB 4.06% github.com/rclone/rclone/vfs.(*File).openRW
0 0% 97.99% 35.49MB 2.40% github.com/rclone/rclone/vfs.(*RWFileHandle).Truncate
0 0% 97.99% 35.49MB 2.40% github.com/rclone/rclone/vfs.(*RWFileHandle).openPending
0 0% 97.99% 62.01MB 4.20% github.com/rclone/rclone/vfs.(*ReadFileHandle).Flush
0 0% 97.99% 62.01MB 4.20% github.com/rclone/rclone/vfs.(*ReadFileHandle).checkHash
0 0% 97.99% 99.47MB 6.74% github.com/rclone/rclone/vfs/vfscache.(*Cache).Exists
0 0% 97.99% 99.47MB 6.74% github.com/rclone/rclone/vfs/vfscache.(*Cache).get
0 0% 97.99% 21.50MB 1.46% github.com/rclone/rclone/vfs/vfscache.(*Item).Close.func2
0 0% 97.99% 21.50MB 1.46% github.com/rclone/rclone/vfs/vfscache.(*Item)._store
0 0% 97.99% 21.50MB 1.46% github.com/rclone/rclone/vfs/vfscache.(*Item).store
0 0% 97.99% 21.50MB 1.46% github.com/rclone/rclone/vfs/vfscache/writeback.(*WriteBack).upload
0 0% 97.99% 157.54MB 10.67% google.golang.org/api/internal/gensupport.DecodeResponse (inline)
0 0% 97.99% 464MB 31.44% google.golang.org/api/internal/gensupport.NewInfoFromMedia
0 0% 97.99% 464MB 31.44% google.golang.org/api/internal/gensupport.PrepareUpload
0 0% 97.99% 49.52MB 3.36% google.golang.org/api/storage/v1.(*ObjectsInsertCall).Do
0 0% 97.99% 464MB 31.44% google.golang.org/api/storage/v1.(*ObjectsInsertCall).Media
0 0% 97.99% 108.53MB 7.35% google.golang.org/api/storage/v1.(*ObjectsListCall).Do
0 0% 97.99% 47MB 3.18% path/filepath.Join (inline)
0 0% 97.99% 47MB 3.18% path/filepath.join
0 0% 97.99% 47MB 3.18% strings.(*Builder).Grow (inline)
0 0% 97.99% 47MB 3.18% strings.Join
Fetching profile over HTTP from http://localhost:5572/debug/pprof/heap
Saved profile in /root/pprof/pprof.rclone.alloc_objects.alloc_space.inuse_objects.inuse_space.003.pb.gz
File: rclone
Type: inuse_space
Time: Jul 22, 2022 at 6:30pm (-03)
Showing nodes accounting for 2951.86MB, 98.23% of 3004.93MB total
Dropped 173 nodes (cum <= 15.02MB)
flat flat% sum% cum cum%
608MB 20.23% 20.23% 608MB 20.23% google.golang.org/api/internal/gensupport.NewMediaBuffer (inline)
375.55MB 12.50% 32.73% 375.55MB 12.50% encoding/json.(*decodeState).literalStore
283.05MB 9.42% 42.15% 283.05MB 9.42% github.com/rclone/rclone/vfs.newFile
203.03MB 6.76% 48.91% 230.54MB 7.67% github.com/rclone/rclone/fs/hash.(*MultiHasher).Sums
174.54MB 5.81% 54.72% 237.54MB 7.91% github.com/rclone/rclone/vfs/vfscache.newItem
141.51MB 4.71% 59.43% 242.52MB 8.07% github.com/rclone/rclone/backend/local.(*Fs).newObject
113.52MB 3.78% 63.20% 128.52MB 4.28% github.com/rclone/rclone/fs/hash.StreamTypes
105.01MB 3.49% 66.70% 892.54MB 29.70% github.com/rclone/rclone/backend/googlecloudstorage.(*Fs).Put
100.51MB 3.34% 70.04% 100.51MB 3.34% strings.(*Builder).grow (inline)
100.01MB 3.33% 73.37% 100.01MB 3.33% path.Join
98.47MB 3.28% 76.65% 98.47MB 3.28% github.com/rclone/rclone/vfs.(*Dir).addObject
96MB 3.19% 79.84% 96MB 3.19% encoding/hex.EncodeToString (inline)
80.51MB 2.68% 82.52% 102.51MB 3.41% github.com/rclone/rclone/backend/googlecloudstorage.(*Fs).newObjectWithInfo
79MB 2.63% 85.15% 79MB 2.63% github.com/rclone/rclone/backend/union.(*Fs).wrapEntries
68.50MB 2.28% 87.43% 68.50MB 2.28% github.com/rclone/rclone/backend/union/upstream.(*Fs).WrapObject (inline)
63MB 2.10% 89.53% 63MB 2.10% sync.NewCond (inline)
56.50MB 1.88% 91.41% 162.94MB 5.42% github.com/rclone/rclone/vfs.newRWFileHandle
56.22MB 1.87% 93.28% 228.25MB 7.60% github.com/rclone/rclone/vfs.(*Dir)._readDirFromEntries
32.51MB 1.08% 94.36% 32.51MB 1.08% github.com/rclone/rclone/vfs.newDir
28.50MB 0.95% 95.31% 67.50MB 2.25% github.com/rclone/rclone/backend/union.(*Fs).mergeDirEntries
22.45MB 0.75% 96.06% 22.45MB 0.75% bazil.org/fuse/fs.(*Server).saveNode
20.50MB 0.68% 96.74% 61MB 2.03% github.com/rclone/rclone/backend/union.(*Fs).put
18MB 0.6% 97.34% 18MB 0.6% github.com/rclone/rclone/fs.NewDir (inline)
12.50MB 0.42% 97.75% 308.25MB 10.26% github.com/rclone/rclone/cmd/mount.(*Dir).Lookup
10MB 0.33% 98.09% 526.50MB 17.52% github.com/rclone/rclone/cmd/mount.(*Dir).Create
3.96MB 0.13% 98.22% 241.50MB 8.04% github.com/rclone/rclone/vfs/vfscache.(*Cache)._get
0.50MB 0.017% 98.23% 166.53MB 5.54% google.golang.org/api/storage/v1.(*ObjectsInsertCall).Do
0 0% 98.23% 1076.85MB 35.84% bazil.org/fuse/fs.(*Server).Serve.func1
0 0% 98.23% 1076.85MB 35.84% bazil.org/fuse/fs.(*Server).handleRequest
0 0% 98.23% 22.95MB 0.76% bazil.org/fuse/fs.(*Server).saveLookup
0 0% 98.23% 1076.85MB 35.84% bazil.org/fuse/fs.(*Server).serve
0 0% 98.23% 375.55MB 12.50% encoding/json.(*Decoder).Decode
0 0% 98.23% 210.03MB 6.99% encoding/json.(*decodeState).array
0 0% 98.23% 375.55MB 12.50% encoding/json.(*decodeState).object
0 0% 98.23% 375.55MB 12.50% encoding/json.(*decodeState).unmarshal
0 0% 98.23% 375.55MB 12.50% encoding/json.(*decodeState).value
0 0% 98.23% 326.04MB 10.85% github.com/rclone/rclone/backend/googlecloudstorage.(*Fs).List
0 0% 98.23% 116.01MB 3.86% github.com/rclone/rclone/backend/googlecloudstorage.(*Fs).itemToDirEntry
0 0% 98.23% 326.04MB 10.85% github.com/rclone/rclone/backend/googlecloudstorage.(*Fs).list
0 0% 98.23% 210.03MB 6.99% github.com/rclone/rclone/backend/googlecloudstorage.(*Fs).list.func1
0 0% 98.23% 326.04MB 10.85% github.com/rclone/rclone/backend/googlecloudstorage.(*Fs).listDir
0 0% 98.23% 116.01MB 3.86% github.com/rclone/rclone/backend/googlecloudstorage.(*Fs).listDir.func1
0 0% 98.23% 806.03MB 26.82% github.com/rclone/rclone/backend/googlecloudstorage.(*Object).Update
0 0% 98.23% 774.53MB 25.78% github.com/rclone/rclone/backend/googlecloudstorage.(*Object).Update.func1
0 0% 98.23% 53.50MB 1.78% github.com/rclone/rclone/backend/googlecloudstorage.(*Object).setMetaData
0 0% 98.23% 168.01MB 5.59% github.com/rclone/rclone/backend/local.(*Fs).List
0 0% 98.23% 338.06MB 11.25% github.com/rclone/rclone/backend/local.(*Fs).Put
0 0% 98.23% 49MB 1.63% github.com/rclone/rclone/backend/local.(*Fs).cleanRemote
0 0% 98.23% 101.01MB 3.36% github.com/rclone/rclone/backend/local.(*Fs).localPath
0 0% 98.23% 115.01MB 3.83% github.com/rclone/rclone/backend/local.(*Fs).newObjectWithInfo
0 0% 98.23% 129.02MB 4.29% github.com/rclone/rclone/backend/local.(*Object).Hash
0 0% 98.23% 231.05MB 7.69% github.com/rclone/rclone/backend/local.(*Object).Update
0 0% 98.23% 67.50MB 2.25% github.com/rclone/rclone/backend/union.(*Fs).List
0 0% 98.23% 531.05MB 17.67% github.com/rclone/rclone/backend/union.(*Fs).List.func1
0 0% 98.23% 61MB 2.03% github.com/rclone/rclone/backend/union.(*Fs).Put
0 0% 98.23% 1266.59MB 42.15% github.com/rclone/rclone/backend/union.(*Fs).put.func1
0 0% 98.23% 39.01MB 1.30% github.com/rclone/rclone/backend/union.(*Object).Update.func1
0 0% 98.23% 1837.15MB 61.14% github.com/rclone/rclone/backend/union.multithread.func1
0 0% 98.23% 1230.59MB 40.95% github.com/rclone/rclone/backend/union/upstream.(*Fs).Put
0 0% 98.23% 37MB 1.23% github.com/rclone/rclone/backend/union/upstream.(*Fs).WrapEntry
0 0% 98.23% 39.01MB 1.30% github.com/rclone/rclone/backend/union/upstream.(*Object).Update
0 0% 98.23% 78.46MB 2.61% github.com/rclone/rclone/cmd/mount.(*File).Open
0 0% 98.23% 128.02MB 4.26% github.com/rclone/rclone/cmd/mount.(*FileHandle).Flush
0 0% 98.23% 984.55MB 32.76% github.com/rclone/rclone/fs.pacerInvoker
0 0% 98.23% 67.50MB 2.25% github.com/rclone/rclone/fs/list.DirSorted
0 0% 98.23% 62MB 2.06% github.com/rclone/rclone/fs/operations.Copy
0 0% 98.23% 210.03MB 6.99% github.com/rclone/rclone/lib/pacer.(*Pacer).Call
0 0% 98.23% 774.53MB 25.78% github.com/rclone/rclone/lib/pacer.(*Pacer).CallNoRetry (inline)
0 0% 98.23% 984.55MB 32.76% github.com/rclone/rclone/lib/pacer.(*Pacer).call
0 0% 98.23% 140.03MB 4.66% github.com/rclone/rclone/vfs.(*Dir).Create
0 0% 98.23% 295.75MB 9.84% github.com/rclone/rclone/vfs.(*Dir).Stat
0 0% 98.23% 295.75MB 9.84% github.com/rclone/rclone/vfs.(*Dir)._readDir
0 0% 98.23% 295.75MB 9.84% github.com/rclone/rclone/vfs.(*Dir).stat
0 0% 98.23% 454.94MB 15.14% github.com/rclone/rclone/vfs.(*File).Open
0 0% 98.23% 50.50MB 1.68% github.com/rclone/rclone/vfs.(*File).Path
0 0% 98.23% 162.94MB 5.42% github.com/rclone/rclone/vfs.(*File).openRW
0 0% 98.23% 96.46MB 3.21% github.com/rclone/rclone/vfs.(*RWFileHandle).Truncate
0 0% 98.23% 96.46MB 3.21% github.com/rclone/rclone/vfs.(*RWFileHandle).openPending
0 0% 98.23% 128.02MB 4.26% github.com/rclone/rclone/vfs.(*ReadFileHandle).Flush
0 0% 98.23% 128.52MB 4.28% github.com/rclone/rclone/vfs.(*ReadFileHandle).checkHash
0 0% 98.23% 241.50MB 8.04% github.com/rclone/rclone/vfs/vfscache.(*Cache).Exists
0 0% 98.23% 241.50MB 8.04% github.com/rclone/rclone/vfs/vfscache.(*Cache).get
0 0% 98.23% 64MB 2.13% github.com/rclone/rclone/vfs/vfscache.(*Item).Close.func2
0 0% 98.23% 64MB 2.13% github.com/rclone/rclone/vfs/vfscache.(*Item)._store
0 0% 98.23% 64MB 2.13% github.com/rclone/rclone/vfs/vfscache.(*Item).store
0 0% 98.23% 64MB 2.13% github.com/rclone/rclone/vfs/vfscache/writeback.(*WriteBack).upload
0 0% 98.23% 375.55MB 12.50% google.golang.org/api/internal/gensupport.DecodeResponse (inline)
0 0% 98.23% 608MB 20.23% google.golang.org/api/internal/gensupport.NewInfoFromMedia
0 0% 98.23% 608MB 20.23% google.golang.org/api/internal/gensupport.PrepareUpload
0 0% 98.23% 608MB 20.23% google.golang.org/api/storage/v1.(*ObjectsInsertCall).Media
0 0% 98.23% 210.03MB 6.99% google.golang.org/api/storage/v1.(*ObjectsListCall).Do
0 0% 98.23% 100.51MB 3.34% path/filepath.Join (inline)
0 0% 98.23% 100.51MB 3.34% path/filepath.join
0 0% 98.23% 100.51MB 3.34% strings.(*Builder).Grow (inline)
0 0% 98.23% 100.51MB 3.34% strings.Join
rclone v1.59.0
os/version: ubuntu 20.04 (64 bit)
os/kernel: 5.15.0-1013-gcp (x86_64)
os/type: linux
os/arch: amd64
go/version: go1.18.3
go/linking: static
go/tags: none
google cloud storage
rclone copy /tmp remote:tmp)
rclone mount remote:bucket /remote --use-mmap --disable-http2 --rc
[remote]
type = google cloud storage
service_account_file =
ncw
July 23, 2022, 12:09am
2
That looks like a leak in the Google cloud storage SDK...
Can you make the image version of the profile and post that?
cberni
July 23, 2022, 2:14am
3
yes, I can. which command should I run to create a image version?
I tried to set the environment variable GOGC=20 (default 100) for the garbage collector to be more aggressive. But I don't know if that would help.
ncw
July 23, 2022, 10:28am
4
Doing this should show what I need which is the path to how the memory got allocated
go tool pprof -svg http://localhost:5572/debug/pprof/heap
It might help.
I think looking at the trace, the problem here is that you have lots of items in the directory cache. If you clear the directory cache with rclone rc vfs/forget and compare the memory profiles before and after do you notice a big difference?
1 Like
cberni
July 25, 2022, 1:00pm
5
yes. have a big differece.
attached profile before vfs/forget and after.
you must rename profiles.txt to profiles.zipprofiles.txt (42.4 KB)
we also noticed a bit of high load at this moment that I took the profile.
ncw
July 25, 2022, 5:20pm
6
The memory usage went down from 4G to 800M!
Of that 800M, 500M is in the google cloud storage SDK which looks like upload buffers.
So what is going on here is that the VFS directory cache has lots of objects in it - about 2 million - does that makes sense to you?
This has this parameter to control its size
--dir-cache-time duration Time to cache directory entries for (default 5m0s)
So if you set this lower then rclone will use less memory. You can also run vfs/forget periodically too.
The directory cache doesn't have a total items flag, but it probably should, so you could set --dir-cache-max-items 1000000 say and be sure rclone isn't going to use too much memory.
1 Like
cberni
July 26, 2022, 1:21pm
7
configs yesterdays profile:
I changed --dir-cache-time yesterday to 1h but still got memory increasing (profile attached)
--dir-cache-max-items would be a nice feature.
must chante profile003.txt to profile003.zipprofile003.txt (21.3 KB)
Note:
ncw
July 26, 2022, 2:25pm
8
It looks like you have about 1.5M files in the cache.
Does that seem right to you?
How many files do you use on a regular basis?
I think if you need to use less memory, then that feature is needed.
Or alternatively we rework the directory caching to run off disk - this is one of my ideas for increasing the startup time when there are lots of things in the cache.
cberni
July 26, 2022, 2:46pm
9
yesterdays production:
number of files | size | hour
34287 | 6.1G | 0
18310 | 3.3G | 1
11224 | 1.9G | 2
8957 | 1.6G | 3
10497 | 2.0G | 4
7804 | 1.5G | 5
4902 | 871M | 6
25801 | 4.1G | 7
75382 | 12G | 8
97002 | 16G | 9
84661 | 15G | 10
101114 18G | 11
77904 | 15G | 12
58310 | 12G | 13
70879 | 13G | 14
78620 | 15G | 15
94324 | 18G | 16
115236 21G | 17
80565 | 13G | 18
40953 | 8.0G | 19
31538 | 5.5G | 20
35090 | 6.3G | 21
35676 | 6.4G | 22
19946 | 3.8G | 23
ncw
July 27, 2022, 8:47am
10
You have a lot of files...
Do the files, once produced get accessed a lot? If not then a --dir-cache-max-items flag would be perfect for you I think.
1 Like
cberni
July 27, 2022, 1:58pm
11
not a lot. once produced it will be accessed once more a day and usually no more.
opened 03:27PM - 26 Jul 22 UTC
#### The associated forum post URL from `https://forum.rclone.org`
https://fo… rum.rclone.org/t/rclone-mount-memory-increasing-linearly/32050
#### What is your current rclone version (output from `rclone version`)?
rclone v1.59.0
- os/version: ubuntu 20.04 (64 bit)
- os/kernel: 5.15.0-1013-gcp (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.18.3
- go/linking: static
- go/tags: none
#### What problem are you are trying to solve?
rclone mount memory increasing linearly
#### How do you think rclone should be changed to solve that?
create the flag `--dir-cache-max-items` as a way to limit and help the memory use
#### How to use GitHub
* Please use the 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to show that you are affected by the same issue.
* Please don't comment if you have no relevant information to add. It's just extra noise for everyone subscribed to this issue.
* Subscribe to receive notifications on status change and new comments.
cberni
July 27, 2022, 2:44pm
12
@ncw
yesterday night I changed the virtual machine vCPU from 6 to 8 and it seens work better on memory release.
go tool pprof -text http://localhost:5572/debug/pprof/profile
Fetching profile over HTTP from http://localhost:5572/debug/pprof/profile
Saved profile in /root/pprof/pprof.rclone.samples.cpu.002.pb.gz
File: rclone
Type: cpu
Time: Jul 27, 2022 at 11:37am (-03)
Duration: 30.18s, Total samples = 119.89s (397.24%)
Showing nodes accounting for 112.23s, 93.61% of 119.89s total
Dropped 769 nodes (cum <= 0.60s)
flat flat% sum% cum cum%
29.09s 24.26% 24.26% 31.76s 26.49% runtime.findObject
26.53s 22.13% 46.39% 104s 86.75% runtime.scanobject
14.57s 12.15% 58.55% 14.57s 12.15% runtime.heapBits.bits (inline)
11.88s 9.91% 68.45% 11.88s 9.91% runtime.markBits.isMarked (inline)
7.06s 5.89% 74.34% 7.06s 5.89% runtime.(*gcWork).putFast (inline)
6.07s 5.06% 79.41% 6.19s 5.16% runtime.pageIndexOf
3.45s 2.88% 82.28% 3.45s 2.88% runtime.memclrNoHeapPointers
2.53s 2.11% 84.39% 28.25s 23.56% runtime.greyobject
1.45s 1.21% 85.60% 1.47s 1.23% syscall.Syscall
1.26s 1.05% 86.65% 1.31s 1.09% runtime.heapBitsForAddr
1.24s 1.03% 87.69% 1.27s 1.06% runtime.heapBits.next
1.24s 1.03% 88.72% 1.33s 1.11% runtime.spanOf
0.98s 0.82% 89.54% 56.58s 47.19% runtime.gcDrainN
0.87s 0.73% 90.27% 0.88s 0.73% syscall.Syscall6
0.85s 0.71% 90.98% 0.85s 0.71% runtime.(*mspan).divideByElemSize
0.74s 0.62% 91.59% 49.78s 41.52% runtime.gcDrain
0.63s 0.53% 92.12% 2.42s 2.02% runtime.sweepone
0.61s 0.51% 92.63% 0.61s 0.51% runtime.futex
0.41s 0.34% 92.97% 0.61s 0.51% runtime.(*gcBits).bitp (inline)
0.29s 0.24% 93.21% 1.14s 0.95% runtime.(*mspan).objIndex
0.16s 0.13% 93.34% 0.77s 0.64% runtime.(*sweepLocked).sweep
0.08s 0.067% 93.41% 1.08s 0.9% runtime.schedule
0.06s 0.05% 93.46% 62.02s 51.73% runtime.mallocgc
0.03s 0.025% 93.49% 4.11s 3.43% bazil.org/fuse/fs.(*Server).serve
0.03s 0.025% 93.51% 1.48s 1.23% runtime.newobject
0.02s 0.017% 93.53% 0.66s 0.55% bazil.org/fuse/fs.(*Server).Serve
0.02s 0.017% 93.54% 1.07s 0.89% runtime.(*mcentral).cacheSpan
0.02s 0.017% 93.56% 1.37s 1.14% runtime.deductSweepCredit
0.02s 0.017% 93.58% 1.24s 1.03% runtime.mcall
0.01s 0.0083% 93.59% 59.17s 49.35% github.com/rclone/rclone/backend/googlecloudstorage.(*Object).Update.func1
0.01s 0.0083% 93.59% 59.38s 49.53% github.com/rclone/rclone/lib/pacer.(*Pacer).call
0.01s 0.0083% 93.60% 56.59s 47.20% runtime.gcAssistAlloc
0.01s 0.0083% 93.61% 0.67s 0.56% runtime.newstack
0 0% 93.61% 4.12s 3.44% bazil.org/fuse/fs.(*Server).Serve.func1
0 0% 93.61% 3.69s 3.08% bazil.org/fuse/fs.(*Server).handleRequest
0 0% 93.61% 59.24s 49.41% github.com/rclone/rclone/backend/googlecloudstorage.(*Fs).Put
0 0% 93.61% 59.22s 49.40% github.com/rclone/rclone/backend/googlecloudstorage.(*Object).Update
0 0% 93.61% 0.81s 0.68% github.com/rclone/rclone/backend/local.(*Fs).Put
0 0% 93.61% 0.72s 0.6% github.com/rclone/rclone/backend/local.(*Object).Update
0 0% 93.61% 60.05s 50.09% github.com/rclone/rclone/backend/union.(*Fs).put.func1
0 0% 93.61% 0.71s 0.59% github.com/rclone/rclone/backend/union.multiReader.func1
0 0% 93.61% 60.39s 50.37% github.com/rclone/rclone/backend/union.multithread.func1
0 0% 93.61% 60.05s 50.09% github.com/rclone/rclone/backend/union/upstream.(*Fs).Put
0 0% 93.61% 0.70s 0.58% github.com/rclone/rclone/cmd/mount.(*FileHandle).Write
0 0% 93.61% 0.66s 0.55% github.com/rclone/rclone/cmd/mount.mount.func2
0 0% 93.61% 59.35s 49.50% github.com/rclone/rclone/fs.pacerInvoker
0 0% 93.61% 0.65s 0.54% github.com/rclone/rclone/fs/accounting.(*Account).WriteTo
0 0% 93.61% 59.19s 49.37% github.com/rclone/rclone/lib/pacer.(*Pacer).CallNoRetry (inline)
0 0% 93.61% 0.66s 0.55% github.com/rclone/rclone/vfs.(*RWFileHandle).WriteAt
0 0% 93.61% 0.66s 0.55% github.com/rclone/rclone/vfs.(*RWFileHandle)._writeAt
0 0% 93.61% 58.90s 49.13% google.golang.org/api/internal/gensupport.NewInfoFromMedia
0 0% 93.61% 58.80s 49.04% google.golang.org/api/internal/gensupport.NewMediaBuffer (inline)
0 0% 93.61% 58.90s 49.13% google.golang.org/api/internal/gensupport.PrepareUpload
0 0% 93.61% 58.90s 49.13% google.golang.org/api/storage/v1.(*ObjectsInsertCall).Media
0 0% 93.61% 1.59s 1.33% io.Copy (inline)
0 0% 93.61% 1.59s 1.33% io.copyBuffer
0 0% 93.61% 1.10s 0.92% runtime.(*mcache).nextFree
0 0% 93.61% 1.09s 0.91% runtime.(*mcache).refill
0 0% 93.61% 0.61s 0.51% runtime.(*mspan).markBitsForIndex (inline)
0 0% 93.61% 0.95s 0.79% runtime.bgsweep
0 0% 93.61% 56.58s 47.19% runtime.gcAssistAlloc.func1
0 0% 93.61% 56.58s 47.19% runtime.gcAssistAlloc1
0 0% 93.61% 49.78s 41.52% runtime.gcBgMarkWorker
0 0% 93.61% 49.78s 41.52% runtime.gcBgMarkWorker.func2
0 0% 93.61% 60.19s 50.20% runtime.makeslice
0 0% 93.61% 3.31s 2.76% runtime.memclrNoHeapPointersChunked
0 0% 93.61% 107.18s 89.40% runtime.systemstack
0 0% 93.61% 0.74s 0.62% syscall.Write (inline)
0 0% 93.61% 0.74s 0.62% syscall.write
any way to tune cpu usage?
ncw
July 27, 2022, 5:11pm
13
All those things in the profile are the garbage collector!
Did you set GOGC=20 ? That will use more CPU. You could try resetting it to 100 or use a larger value than 20?
I think what we need to do is implement the --dir-cache-max-items flag and this will fix the memory growth and the CPU usage from garbage collection.
There are probably some easy wins in the vfs cache also to reduce memory usage.
1 Like
cberni
July 28, 2022, 1:38am
14
after change GOGC=20 to GOGC=100 (default).
ncw
July 28, 2022, 5:37pm
15
OK, that's good!
Can you have a go with this? Which will use less memory - I went through the VFS and fixed some easy wins. It might make 10% difference in memory usage but it will make 50% less objects which will be noticeable for less garbage collection time.
v1.60.0-beta.6378.c9619d16d.fix-vfs-startup on branch fix-vfs-startup (uploaded in 15-30 mins)
1 Like
cberni
July 29, 2022, 2:14pm
16
changed to 1.60.0-beta yesterday night and it seens work much better today.
I'm using --dir-cache-time = 12h right now. I'll test it with 1h on monday. Last time I changed it to 1h the mount stuck twice a day.
must chante profile004.txt to profile004.zipprofile004.txt (21.9 KB)
--dir-cache-max-items would respect --dir-cache-time?
are you fixing the issue Reduce startup time with vfs cache (writes/full) ? I tested here and the problem remains.
ncw
July 29, 2022, 5:09pm
17
Great!
Yes it would respect both.
I've been coming up with a plan which I'll discuss in the other thread.
1 Like
cberni
July 30, 2022, 12:58am
18
green cpu and orange memory are rclone mount.--dir-cache-time = 12h right now. On monday I'll test it with 1h.
Fetching profile over HTTP from http://localhost:5572/debug/pprof/heap
Saved profile in /root/pprof/pprof.rclone.alloc_objects.alloc_space.inuse_objects.inuse_space.009.pb.gz
File: rclone
Type: inuse_space
Time: Jul 29, 2022 at 8:01pm (-03)
Showing nodes accounting for 4035.26MB, 98.54% of 4095.15MB total
Dropped 162 nodes (cum <= 20.48MB)
flat flat% sum% cum cum%
896MB 21.88% 21.88% 896MB 21.88% google.golang.org/api/internal/gensupport.NewMediaBuffer (inline)
487.07MB 11.89% 33.77% 487.07MB 11.89% encoding/json.(*decodeState).literalStore
396.07MB 9.67% 43.44% 396.07MB 9.67% github.com/rclone/rclone/vfs.newFile
288.07MB 7.03% 50.48% 288.07MB 7.03% github.com/rclone/rclone/vfs/vfscache.newItem
271.55MB 6.63% 57.11% 305.55MB 7.46% github.com/rclone/rclone/fs/hash.(*MultiHasher).Sums
187.02MB 4.57% 61.68% 318.03MB 7.77% github.com/rclone/rclone/backend/local.(*Fs).newObject
169.03MB 4.13% 65.80% 185.53MB 4.53% github.com/rclone/rclone/fs/hash.StreamTypes
140.01MB 3.42% 69.22% 140.01MB 3.42% path.Join
131.01MB 3.20% 72.42% 131.01MB 3.20% strings.(*Builder).grow (inline)
122.50MB 2.99% 75.41% 122.50MB 2.99% encoding/hex.EncodeToString (inline)
115.01MB 2.81% 78.22% 148.51MB 3.63% github.com/rclone/rclone/backend/googlecloudstorage.(*Fs).newObjectWithInfo
114.85MB 2.80% 81.03% 114.85MB 2.80% github.com/rclone/rclone/vfs.(*Dir).addObject
114.51MB 2.80% 83.82% 1245.04MB 30.40% github.com/rclone/rclone/backend/googlecloudstorage.(*Fs).Put
107.50MB 2.63% 86.45% 107.50MB 2.63% github.com/rclone/rclone/backend/union.(*Fs).wrapEntries
98.30MB 2.40% 88.85% 325.84MB 7.96% github.com/rclone/rclone/vfs.(*Dir)._readDirFromEntries
82MB 2.00% 90.85% 82MB 2.00% github.com/rclone/rclone/backend/union/upstream.(*Fs).WrapObject (inline)
62.50MB 1.53% 92.38% 187.33MB 4.57% github.com/rclone/rclone/vfs.newRWFileHandle
39.51MB 0.96% 93.34% 39.51MB 0.96% github.com/rclone/rclone/vfs.newDir
38.50MB 0.94% 94.28% 94.50MB 2.31% github.com/rclone/rclone/backend/union.(*Fs).mergeDirEntries
38MB 0.93% 95.21% 91.01MB 2.22% github.com/rclone/rclone/backend/union.(*Fs).put
23.44MB 0.57% 95.78% 23.44MB 0.57% bazil.org/fuse/fs.(*Server).saveNode
22.19MB 0.54% 96.32% 22.19MB 0.54% bazil.org/fuse.newBuffer (inline)
21.50MB 0.53% 96.85% 21.50MB 0.53% github.com/rclone/rclone/fs.NewDir (inline)
20.89MB 0.51% 97.36% 21.89MB 0.53% github.com/rclone/rclone/cmd/mount.(*FileHandle).Read
17.26MB 0.42% 97.78% 1491.81MB 36.43% bazil.org/fuse/fs.(*Server).handleRequest
12MB 0.29% 98.07% 683.44MB 16.69% github.com/rclone/rclone/cmd/mount.(*Dir).Create
11MB 0.27% 98.34% 431.35MB 10.53% github.com/rclone/rclone/cmd/mount.(*Dir).Lookup
7.96MB 0.19% 98.54% 296.03MB 7.23% github.com/rclone/rclone/vfs/vfscache.(*Cache)._get
0 0% 98.54% 22.19MB 0.54% bazil.org/fuse.(*ReadRequest).Respond
0 0% 98.54% 1492.81MB 36.45% bazil.org/fuse/fs.(*Server).Serve.func1
0 0% 98.54% 23.44MB 0.57% bazil.org/fuse/fs.(*Server).saveLookup
0 0% 98.54% 1492.81MB 36.45% bazil.org/fuse/fs.(*Server).serve
0 0% 98.54% 487.07MB 11.89% encoding/json.(*Decoder).Decode
0 0% 98.54% 275.53MB 6.73% encoding/json.(*decodeState).array
0 0% 98.54% 487.07MB 11.89% encoding/json.(*decodeState).object
0 0% 98.54% 487.07MB 11.89% encoding/json.(*decodeState).unmarshal
0 0% 98.54% 487.07MB 11.89% encoding/json.(*decodeState).value
0 0% 98.54% 435.05MB 10.62% github.com/rclone/rclone/backend/googlecloudstorage.(*Fs).List
0 0% 98.54% 159.51MB 3.90% github.com/rclone/rclone/backend/googlecloudstorage.(*Fs).itemToDirEntry
0 0% 98.54% 435.05MB 10.62% github.com/rclone/rclone/backend/googlecloudstorage.(*Fs).list
0 0% 98.54% 275.53MB 6.73% github.com/rclone/rclone/backend/googlecloudstorage.(*Fs).list.func1
0 0% 98.54% 435.05MB 10.62% github.com/rclone/rclone/backend/googlecloudstorage.(*Fs).listDir
0 0% 98.54% 159.51MB 3.90% github.com/rclone/rclone/backend/googlecloudstorage.(*Fs).listDir.func1
0 0% 98.54% 1146.53MB 28.00% github.com/rclone/rclone/backend/googlecloudstorage.(*Object).Update
0 0% 98.54% 1107.53MB 27.04% github.com/rclone/rclone/backend/googlecloudstorage.(*Object).Update.func1
0 0% 98.54% 72MB 1.76% github.com/rclone/rclone/backend/googlecloudstorage.(*Object).setMetaData
0 0% 98.54% 227.52MB 5.56% github.com/rclone/rclone/backend/local.(*Fs).List
0 0% 98.54% 448.64MB 10.96% github.com/rclone/rclone/backend/local.(*Fs).Put
0 0% 98.54% 64MB 1.56% github.com/rclone/rclone/backend/local.(*Fs).cleanRemote
0 0% 98.54% 131.01MB 3.20% github.com/rclone/rclone/backend/local.(*Fs).localPath
0 0% 98.54% 155.51MB 3.80% github.com/rclone/rclone/backend/local.(*Fs).newObjectWithInfo
0 0% 98.54% 185.53MB 4.53% github.com/rclone/rclone/backend/local.(*Object).Hash
0 0% 98.54% 308.13MB 7.52% github.com/rclone/rclone/backend/local.(*Object).Update
0 0% 98.54% 94.50MB 2.31% github.com/rclone/rclone/backend/union.(*Fs).List
0 0% 98.54% 706.07MB 17.24% github.com/rclone/rclone/backend/union.(*Fs).List.func1
0 0% 98.54% 91.01MB 2.22% github.com/rclone/rclone/backend/union.(*Fs).Put
0 0% 98.54% 1735.18MB 42.37% github.com/rclone/rclone/backend/union.(*Fs).put.func1
0 0% 98.54% 38.01MB 0.93% github.com/rclone/rclone/backend/union.(*Object).Update.func1
0 0% 98.54% 2479.75MB 60.55% github.com/rclone/rclone/backend/union.multithread.func1
0 0% 98.54% 1693.68MB 41.36% github.com/rclone/rclone/backend/union/upstream.(*Fs).Put
0 0% 98.54% 43.50MB 1.06% github.com/rclone/rclone/backend/union/upstream.(*Fs).WrapEntry
0 0% 98.54% 38.01MB 0.93% github.com/rclone/rclone/backend/union/upstream.(*Object).Update
0 0% 98.54% 88.47MB 2.16% github.com/rclone/rclone/cmd/mount.(*File).Open
0 0% 98.54% 185.53MB 4.53% github.com/rclone/rclone/cmd/mount.(*FileHandle).Flush
0 0% 98.54% 1383.57MB 33.79% github.com/rclone/rclone/fs.pacerInvoker
0 0% 98.54% 94.50MB 2.31% github.com/rclone/rclone/fs/list.DirSorted
0 0% 98.54% 91.01MB 2.22% github.com/rclone/rclone/fs/operations.Copy
0 0% 98.54% 276.03MB 6.74% github.com/rclone/rclone/lib/pacer.(*Pacer).Call
0 0% 98.54% 1107.53MB 27.04% github.com/rclone/rclone/lib/pacer.(*Pacer).CallNoRetry (inline)
0 0% 98.54% 1383.57MB 33.79% github.com/rclone/rclone/lib/pacer.(*Pacer).call
0 0% 98.54% 201.54MB 4.92% github.com/rclone/rclone/vfs.(*Dir).Create
0 0% 98.54% 420.35MB 10.26% github.com/rclone/rclone/vfs.(*Dir).Stat
0 0% 98.54% 420.35MB 10.26% github.com/rclone/rclone/vfs.(*Dir)._readDir
0 0% 98.54% 420.35MB 10.26% github.com/rclone/rclone/vfs.(*Dir).stat
0 0% 98.54% 558.36MB 13.63% github.com/rclone/rclone/vfs.(*File).Open
0 0% 98.54% 75MB 1.83% github.com/rclone/rclone/vfs.(*File).Path
0 0% 98.54% 187.33MB 4.57% github.com/rclone/rclone/vfs.(*File).openRW
0 0% 98.54% 110.35MB 2.69% github.com/rclone/rclone/vfs.(*RWFileHandle).Truncate
0 0% 98.54% 110.35MB 2.69% github.com/rclone/rclone/vfs.(*RWFileHandle).openPending
0 0% 98.54% 185.53MB 4.53% github.com/rclone/rclone/vfs.(*ReadFileHandle).Flush
0 0% 98.54% 185.53MB 4.53% github.com/rclone/rclone/vfs.(*ReadFileHandle).checkHash
0 0% 98.54% 296.03MB 7.23% github.com/rclone/rclone/vfs/vfscache.(*Cache).Exists
0 0% 98.54% 296.03MB 7.23% github.com/rclone/rclone/vfs/vfscache.(*Cache).get
0 0% 98.54% 92.01MB 2.25% github.com/rclone/rclone/vfs/vfscache.(*Item).Close.func2
0 0% 98.54% 92.01MB 2.25% github.com/rclone/rclone/vfs/vfscache.(*Item)._store
0 0% 98.54% 92.01MB 2.25% github.com/rclone/rclone/vfs/vfscache.(*Item).store
0 0% 98.54% 92.01MB 2.25% github.com/rclone/rclone/vfs/vfscache/writeback.(*WriteBack).upload
0 0% 98.54% 487.07MB 11.89% google.golang.org/api/internal/gensupport.DecodeResponse (inline)
0 0% 98.54% 896MB 21.88% google.golang.org/api/internal/gensupport.NewInfoFromMedia
0 0% 98.54% 896MB 21.88% google.golang.org/api/internal/gensupport.PrepareUpload
0 0% 98.54% 211.53MB 5.17% google.golang.org/api/storage/v1.(*ObjectsInsertCall).Do
0 0% 98.54% 896MB 21.88% google.golang.org/api/storage/v1.(*ObjectsInsertCall).Media
0 0% 98.54% 275.53MB 6.73% google.golang.org/api/storage/v1.(*ObjectsListCall).Do
0 0% 98.54% 131.01MB 3.20% path/filepath.Join (inline)
0 0% 98.54% 131.01MB 3.20% path/filepath.join
0 0% 98.54% 131.01MB 3.20% strings.(*Builder).Grow (inline)
0 0% 98.54% 131.01MB 3.20% strings.Join
must change profile005.txt to profile005.zipprofile005.txt (21.6 KB)
cberni
July 30, 2022, 2:11pm
19
todays chart without restart yet.
Fetching profile over HTTP from http://localhost:5572/debug/pprof/heap
Saved profile in /root/pprof/pprof.rclone.alloc_objects.alloc_space.inuse_objects.inuse_space.013.pb.gz
File: rclone
Type: inuse_space
Time: Jul 30, 2022 at 11:08am (-03)
Showing nodes accounting for 4921.61MB, 97.57% of 5044.17MB total
Dropped 213 nodes (cum <= 25.22MB)
flat flat% sum% cum cum%
1184MB 23.47% 23.47% 1184MB 23.47% google.golang.org/api/internal/gensupport.NewMediaBuffer (inline)
592.58MB 11.75% 35.22% 592.58MB 11.75% encoding/json.(*decodeState).literalStore
470.59MB 9.33% 44.55% 470.59MB 9.33% github.com/rclone/rclone/vfs.newFile
384.09MB 7.61% 52.16% 384.09MB 7.61% github.com/rclone/rclone/vfs/vfscache.newItem
358.56MB 7.11% 59.27% 404.56MB 8.02% github.com/rclone/rclone/fs/hash.(*MultiHasher).Sums
235.02MB 4.66% 63.93% 399.53MB 7.92% github.com/rclone/rclone/backend/local.(*Fs).newObject
184.53MB 3.66% 67.59% 203.53MB 4.04% github.com/rclone/rclone/fs/hash.StreamTypes
167.01MB 3.31% 70.90% 167.01MB 3.31% path.Join
164.01MB 3.25% 74.15% 164.01MB 3.25% strings.(*Builder).grow (inline)
155.50MB 3.08% 77.24% 155.50MB 3.08% encoding/hex.EncodeToString (inline)
150.52MB 2.98% 80.22% 1635.06MB 32.41% github.com/rclone/rclone/backend/googlecloudstorage.(*Fs).Put
143.30MB 2.84% 83.06% 143.30MB 2.84% github.com/rclone/rclone/vfs.(*Dir).addObject
133.51MB 2.65% 85.71% 177.02MB 3.51% github.com/rclone/rclone/backend/googlecloudstorage.(*Fs).newObjectWithInfo
126.01MB 2.50% 88.21% 126.01MB 2.50% github.com/rclone/rclone/backend/union.(*Fs).wrapEntries
105.39MB 2.09% 90.30% 357.44MB 7.09% github.com/rclone/rclone/vfs.(*Dir)._readDirFromEntries
101MB 2.00% 92.30% 101MB 2.00% github.com/rclone/rclone/backend/union/upstream.(*Fs).WrapObject (inline)
90.01MB 1.78% 94.08% 246.28MB 4.88% github.com/rclone/rclone/vfs.newRWFileHandle
48.01MB 0.95% 95.03% 48.01MB 0.95% github.com/rclone/rclone/vfs.newDir
47.50MB 0.94% 95.98% 113MB 2.24% github.com/rclone/rclone/backend/union.(*Fs).put
44MB 0.87% 96.85% 104.50MB 2.07% github.com/rclone/rclone/backend/union.(*Fs).mergeDirEntries
17MB 0.34% 97.18% 886.91MB 17.58% github.com/rclone/rclone/cmd/mount.(*Dir).Create
11.50MB 0.23% 97.41% 473.44MB 9.39% github.com/rclone/rclone/cmd/mount.(*Dir).Lookup
7.96MB 0.16% 97.57% 392.05MB 7.77% github.com/rclone/rclone/vfs/vfscache.(*Cache)._get
0 0% 97.57% 1734.09MB 34.38% bazil.org/fuse/fs.(*Server).Serve.func1
0 0% 97.57% 1734.09MB 34.38% bazil.org/fuse/fs.(*Server).handleRequest
0 0% 97.57% 1734.09MB 34.38% bazil.org/fuse/fs.(*Server).serve
0 0% 97.57% 592.58MB 11.75% encoding/json.(*Decoder).Decode
0 0% 97.57% 318.04MB 6.31% encoding/json.(*decodeState).array
0 0% 97.57% 592.58MB 11.75% encoding/json.(*decodeState).object
0 0% 97.57% 592.58MB 11.75% encoding/json.(*decodeState).unmarshal
0 0% 97.57% 592.58MB 11.75% encoding/json.(*decodeState).value
0 0% 97.57% 506.56MB 10.04% github.com/rclone/rclone/backend/googlecloudstorage.(*Fs).List
0 0% 97.57% 188.52MB 3.74% github.com/rclone/rclone/backend/googlecloudstorage.(*Fs).itemToDirEntry
0 0% 97.57% 506.56MB 10.04% github.com/rclone/rclone/backend/googlecloudstorage.(*Fs).list
0 0% 97.57% 318.04MB 6.31% github.com/rclone/rclone/backend/googlecloudstorage.(*Fs).list.func1
0 0% 97.57% 506.56MB 10.04% github.com/rclone/rclone/backend/googlecloudstorage.(*Fs).listDir
0 0% 97.57% 188.52MB 3.74% github.com/rclone/rclone/backend/googlecloudstorage.(*Fs).listDir.func1
0 0% 97.57% 1508.04MB 29.90% github.com/rclone/rclone/backend/googlecloudstorage.(*Object).Update
0 0% 97.57% 1459.54MB 28.94% github.com/rclone/rclone/backend/googlecloudstorage.(*Object).Update.func1
0 0% 97.57% 90.50MB 1.79% github.com/rclone/rclone/backend/googlecloudstorage.(*Object).setMetaData
0 0% 97.57% 255.02MB 5.06% github.com/rclone/rclone/backend/local.(*Fs).List
0 0% 97.57% 604.22MB 11.98% github.com/rclone/rclone/backend/local.(*Fs).Put
0 0% 97.57% 70MB 1.39% github.com/rclone/rclone/backend/local.(*Fs).cleanRemote
0 0% 97.57% 164.51MB 3.26% github.com/rclone/rclone/backend/local.(*Fs).localPath
0 0% 97.57% 177.51MB 3.52% github.com/rclone/rclone/backend/local.(*Fs).newObjectWithInfo
0 0% 97.57% 203.53MB 4.04% github.com/rclone/rclone/backend/local.(*Object).Hash
0 0% 97.57% 408.70MB 8.10% github.com/rclone/rclone/backend/local.(*Object).Update
0 0% 97.57% 104.50MB 2.07% github.com/rclone/rclone/backend/union.(*Fs).List
0 0% 97.57% 811.08MB 16.08% github.com/rclone/rclone/backend/union.(*Fs).List.func1
0 0% 97.57% 113MB 2.24% github.com/rclone/rclone/backend/union.(*Fs).Put
0 0% 97.57% 2293.77MB 45.47% github.com/rclone/rclone/backend/union.(*Fs).put.func1
0 0% 97.57% 50.01MB 0.99% github.com/rclone/rclone/backend/union.(*Object).Update.func1
0 0% 97.57% 3156.36MB 62.57% github.com/rclone/rclone/backend/union.multithread.func1
0 0% 97.57% 2239.27MB 44.39% github.com/rclone/rclone/backend/union/upstream.(*Fs).Put
0 0% 97.57% 49.50MB 0.98% github.com/rclone/rclone/backend/union/upstream.(*Fs).WrapEntry
0 0% 97.57% 50.01MB 0.99% github.com/rclone/rclone/backend/union/upstream.(*Object).Update
0 0% 97.57% 122.47MB 2.43% github.com/rclone/rclone/cmd/mount.(*File).Open
0 0% 97.57% 203.53MB 4.04% github.com/rclone/rclone/cmd/mount.(*FileHandle).Flush
0 0% 97.57% 1779.08MB 35.27% github.com/rclone/rclone/fs.pacerInvoker
0 0% 97.57% 104.50MB 2.07% github.com/rclone/rclone/fs/list.DirSorted
0 0% 97.57% 114.50MB 2.27% github.com/rclone/rclone/fs/operations.Copy
0 0% 97.57% 319.54MB 6.33% github.com/rclone/rclone/lib/pacer.(*Pacer).Call
0 0% 97.57% 1459.54MB 28.94% github.com/rclone/rclone/lib/pacer.(*Pacer).CallNoRetry (inline)
0 0% 97.57% 1779.08MB 35.27% github.com/rclone/rclone/lib/pacer.(*Pacer).call
0 0% 97.57% 258.55MB 5.13% github.com/rclone/rclone/vfs.(*Dir).Create
0 0% 97.57% 461.94MB 9.16% github.com/rclone/rclone/vfs.(*Dir).Stat
0 0% 97.57% 461.94MB 9.16% github.com/rclone/rclone/vfs.(*Dir)._readDir
0 0% 97.57% 461.94MB 9.16% github.com/rclone/rclone/vfs.(*Dir).stat
0 0% 97.57% 733.83MB 14.55% github.com/rclone/rclone/vfs.(*File).Open
0 0% 97.57% 95.51MB 1.89% github.com/rclone/rclone/vfs.(*File).Path
0 0% 97.57% 246.28MB 4.88% github.com/rclone/rclone/vfs.(*File).openRW
0 0% 97.57% 139.30MB 2.76% github.com/rclone/rclone/vfs.(*RWFileHandle).Truncate
0 0% 97.57% 139.30MB 2.76% github.com/rclone/rclone/vfs.(*RWFileHandle).openPending
0 0% 97.57% 203.53MB 4.04% github.com/rclone/rclone/vfs.(*ReadFileHandle).Flush
0 0% 97.57% 203.53MB 4.04% github.com/rclone/rclone/vfs.(*ReadFileHandle).checkHash
0 0% 97.57% 392.05MB 7.77% github.com/rclone/rclone/vfs/vfscache.(*Cache).Exists
0 0% 97.57% 392.05MB 7.77% github.com/rclone/rclone/vfs/vfscache.(*Cache).get
0 0% 97.57% 116.50MB 2.31% github.com/rclone/rclone/vfs/vfscache.(*Item).Close.func2
0 0% 97.57% 116.50MB 2.31% github.com/rclone/rclone/vfs/vfscache.(*Item)._store
0 0% 97.57% 116.50MB 2.31% github.com/rclone/rclone/vfs/vfscache.(*Item).store
0 0% 97.57% 116.50MB 2.31% github.com/rclone/rclone/vfs/vfscache/writeback.(*WriteBack).upload
0 0% 97.57% 592.58MB 11.75% google.golang.org/api/internal/gensupport.DecodeResponse (inline)
0 0% 97.57% 1184MB 23.47% google.golang.org/api/internal/gensupport.NewInfoFromMedia
0 0% 97.57% 1184MB 23.47% google.golang.org/api/internal/gensupport.PrepareUpload
0 0% 97.57% 275.54MB 5.46% google.golang.org/api/storage/v1.(*ObjectsInsertCall).Do
0 0% 97.57% 1184MB 23.47% google.golang.org/api/storage/v1.(*ObjectsInsertCall).Media
0 0% 97.57% 318.04MB 6.31% google.golang.org/api/storage/v1.(*ObjectsListCall).Do
0 0% 97.57% 164.51MB 3.26% path/filepath.Join (inline)
0 0% 97.57% 164.51MB 3.26% path/filepath.join
0 0% 97.57% 164.01MB 3.25% strings.(*Builder).Grow (inline)
0 0% 97.57% 164.01MB 3.25% strings.Join
must change profile006.txt to profile006.zipprofile006.txt (21.5 KB)
cberni
August 1, 2022, 9:41pm
20
todays production with --dir-cache-time=1h.--dir-cache-time=12h for tomorrow.
Fetching profile over HTTP from http://localhost:5572/debug/pprof/heap
Saved profile in /root/pprof/pprof.rclone.alloc_objects.alloc_space.inuse_objects.inuse_space.004.pb.gz
File: rclone
Type: inuse_space
Time: Aug 1, 2022 at 6:37pm (-03)
Showing nodes accounting for 4887.08MB, 97.74% of 4999.94MB total
Dropped 190 nodes (cum <= 25MB)
flat flat% sum% cum cum%
1472MB 29.44% 29.44% 1472MB 29.44% google.golang.org/api/internal/gensupport.NewMediaBuffer (inline)
583.08MB 11.66% 41.10% 583.08MB 11.66% encoding/json.(*decodeState).literalStore
400.57MB 8.01% 49.11% 400.57MB 8.01% github.com/rclone/rclone/vfs.newFile
301.07MB 6.02% 55.14% 301.07MB 6.02% github.com/rclone/rclone/vfs/vfscache.newItem
284.55MB 5.69% 60.83% 316.55MB 6.33% github.com/rclone/rclone/fs/hash.(*MultiHasher).Sums
244.52MB 4.89% 65.72% 380.03MB 7.60% github.com/rclone/rclone/backend/local.(*Fs).newObject
211.54MB 4.23% 69.95% 238.54MB 4.77% github.com/rclone/rclone/fs/hash.StreamTypes
170.51MB 3.41% 73.36% 170.51MB 3.41% path.Join
151.52MB 3.03% 76.39% 194.02MB 3.88% github.com/rclone/rclone/backend/googlecloudstorage.(*Fs).newObjectWithInfo
141MB 2.82% 79.21% 141MB 2.82% encoding/hex.EncodeToString (inline)
136.01MB 2.72% 81.93% 136.01MB 2.72% strings.(*Builder).grow (inline)
128.51MB 2.57% 84.50% 128.51MB 2.57% github.com/rclone/rclone/backend/union.(*Fs).wrapEntries
116.51MB 2.33% 86.83% 1858.55MB 37.17% github.com/rclone/rclone/backend/googlecloudstorage.(*Fs).Put
116.50MB 2.33% 89.16% 116.50MB 2.33% github.com/rclone/rclone/backend/union/upstream.(*Fs).WrapObject (inline)
112.17MB 2.24% 91.40% 112.17MB 2.24% github.com/rclone/rclone/vfs.(*Dir).addObject
85.05MB 1.70% 93.10% 299.09MB 5.98% github.com/rclone/rclone/vfs.(*Dir)._readDirFromEntries
70.50MB 1.41% 94.51% 192.67MB 3.85% github.com/rclone/rclone/vfs.newRWFileHandle
48MB 0.96% 95.47% 119.50MB 2.39% github.com/rclone/rclone/backend/union.(*Fs).mergeDirEntries
38MB 0.76% 96.23% 95MB 1.90% github.com/rclone/rclone/backend/union.(*Fs).put
29MB 0.58% 96.81% 29MB 0.58% github.com/rclone/rclone/vfs.newDir
16MB 0.32% 97.13% 711.23MB 14.22% github.com/rclone/rclone/cmd/mount.(*Dir).Create
14.50MB 0.29% 97.42% 36.70MB 0.73% bazil.org/fuse.(*Conn).ReadRequest
9.50MB 0.19% 97.61% 428.09MB 8.56% github.com/rclone/rclone/cmd/mount.(*Dir).Lookup
6.46MB 0.13% 97.74% 307.53MB 6.15% github.com/rclone/rclone/vfs/vfscache.(*Cache)._get
0 0% 97.74% 36.70MB 0.73% bazil.org/fuse/fs.(*Server).Serve
0 0% 97.74% 1493.52MB 29.87% bazil.org/fuse/fs.(*Server).Serve.func1
0 0% 97.74% 1493.52MB 29.87% bazil.org/fuse/fs.(*Server).handleRequest
0 0% 97.74% 1493.52MB 29.87% bazil.org/fuse/fs.(*Server).serve
0 0% 97.74% 583.08MB 11.66% encoding/json.(*Decoder).Decode
0 0% 97.74% 341.04MB 6.82% encoding/json.(*decodeState).array
0 0% 97.74% 583.08MB 11.66% encoding/json.(*decodeState).object
0 0% 97.74% 583.08MB 11.66% encoding/json.(*decodeState).unmarshal
0 0% 97.74% 583.08MB 11.66% encoding/json.(*decodeState).value
0 0% 97.74% 547.56MB 10.95% github.com/rclone/rclone/backend/googlecloudstorage.(*Fs).List
0 0% 97.74% 206.52MB 4.13% github.com/rclone/rclone/backend/googlecloudstorage.(*Fs).itemToDirEntry
0 0% 97.74% 547.56MB 10.95% github.com/rclone/rclone/backend/googlecloudstorage.(*Fs).list
0 0% 97.74% 341.04MB 6.82% github.com/rclone/rclone/backend/googlecloudstorage.(*Fs).list.func1
0 0% 97.74% 547.56MB 10.95% github.com/rclone/rclone/backend/googlecloudstorage.(*Fs).listDir
0 0% 97.74% 206.52MB 4.13% github.com/rclone/rclone/backend/googlecloudstorage.(*Fs).listDir.func1
0 0% 97.74% 1757.54MB 35.15% github.com/rclone/rclone/backend/googlecloudstorage.(*Object).Update
0 0% 97.74% 1716.54MB 34.33% github.com/rclone/rclone/backend/googlecloudstorage.(*Object).Update.func1
0 0% 97.74% 82MB 1.64% github.com/rclone/rclone/backend/googlecloudstorage.(*Object).setMetaData
0 0% 97.74% 301.02MB 6.02% github.com/rclone/rclone/backend/local.(*Fs).List
0 0% 97.74% 490.17MB 9.80% github.com/rclone/rclone/backend/local.(*Fs).Put
0 0% 97.74% 96.01MB 1.92% github.com/rclone/rclone/backend/local.(*Fs).cleanRemote
0 0% 97.74% 135.51MB 2.71% github.com/rclone/rclone/backend/local.(*Fs).localPath
0 0% 97.74% 196.02MB 3.92% github.com/rclone/rclone/backend/local.(*Fs).newObjectWithInfo
0 0% 97.74% 238.54MB 4.77% github.com/rclone/rclone/backend/local.(*Object).Hash
0 0% 97.74% 321.66MB 6.43% github.com/rclone/rclone/backend/local.(*Object).Update
0 0% 97.74% 119.50MB 2.39% github.com/rclone/rclone/backend/union.(*Fs).List
0 0% 97.74% 913.59MB 18.27% github.com/rclone/rclone/backend/union.(*Fs).List.func1
0 0% 97.74% 95.50MB 1.91% github.com/rclone/rclone/backend/union.(*Fs).Put
0 0% 97.74% 2403.22MB 48.07% github.com/rclone/rclone/backend/union.(*Fs).put.func1
0 0% 97.74% 31MB 0.62% github.com/rclone/rclone/backend/union.(*Object).Update.func1
0 0% 97.74% 3349.31MB 66.99% github.com/rclone/rclone/backend/union.multithread.func1
0 0% 97.74% 2348.72MB 46.97% github.com/rclone/rclone/backend/union/upstream.(*Fs).Put
0 0% 97.74% 65MB 1.30% github.com/rclone/rclone/backend/union/upstream.(*Fs).WrapEntry
0 0% 97.74% 31MB 0.62% github.com/rclone/rclone/backend/union/upstream.(*Object).Update
0 0% 97.74% 91.01MB 1.82% github.com/rclone/rclone/cmd/mount.(*File).Open
0 0% 97.74% 238.04MB 4.76% github.com/rclone/rclone/cmd/mount.(*FileHandle).Flush
0 0% 97.74% 36.70MB 0.73% github.com/rclone/rclone/cmd/mount.mount.func2
0 0% 97.74% 2059.08MB 41.18% github.com/rclone/rclone/fs.pacerInvoker
0 0% 97.74% 119.50MB 2.39% github.com/rclone/rclone/fs/list.DirSorted
0 0% 97.74% 96MB 1.92% github.com/rclone/rclone/fs/operations.Copy
0 0% 97.74% 342.54MB 6.85% github.com/rclone/rclone/lib/pacer.(*Pacer).Call
0 0% 97.74% 1716.54MB 34.33% github.com/rclone/rclone/lib/pacer.(*Pacer).CallNoRetry (inline)
0 0% 97.74% 2059.08MB 41.18% github.com/rclone/rclone/lib/pacer.(*Pacer).call
0 0% 97.74% 212.04MB 4.24% github.com/rclone/rclone/vfs.(*Dir).Create
0 0% 97.74% 418.59MB 8.37% github.com/rclone/rclone/vfs.(*Dir).Stat
0 0% 97.74% 418.59MB 8.37% github.com/rclone/rclone/vfs.(*Dir)._readDir
0 0% 97.74% 418.59MB 8.37% github.com/rclone/rclone/vfs.(*Dir).stat
0 0% 97.74% 574.20MB 11.48% github.com/rclone/rclone/vfs.(*File).Open
0 0% 97.74% 74MB 1.48% github.com/rclone/rclone/vfs.(*File).Path
0 0% 97.74% 192.67MB 3.85% github.com/rclone/rclone/vfs.(*File).openRW
0 0% 97.74% 110.17MB 2.20% github.com/rclone/rclone/vfs.(*RWFileHandle).Truncate
0 0% 97.74% 110.17MB 2.20% github.com/rclone/rclone/vfs.(*RWFileHandle).openPending
0 0% 97.74% 238.04MB 4.76% github.com/rclone/rclone/vfs.(*ReadFileHandle).Flush
0 0% 97.74% 238.54MB 4.77% github.com/rclone/rclone/vfs.(*ReadFileHandle).checkHash
0 0% 97.74% 307.53MB 6.15% github.com/rclone/rclone/vfs/vfscache.(*Cache).Exists
0 0% 97.74% 307.53MB 6.15% github.com/rclone/rclone/vfs/vfscache.(*Cache).get
0 0% 97.74% 99MB 1.98% github.com/rclone/rclone/vfs/vfscache.(*Item).Close.func2
0 0% 97.74% 99MB 1.98% github.com/rclone/rclone/vfs/vfscache.(*Item)._store
0 0% 97.74% 99MB 1.98% github.com/rclone/rclone/vfs/vfscache.(*Item).store
0 0% 97.74% 99MB 1.98% github.com/rclone/rclone/vfs/vfscache/writeback.(*WriteBack).upload
0 0% 97.74% 583.08MB 11.66% google.golang.org/api/internal/gensupport.DecodeResponse (inline)
0 0% 97.74% 1472MB 29.44% google.golang.org/api/internal/gensupport.NewInfoFromMedia
0 0% 97.74% 1472MB 29.44% google.golang.org/api/internal/gensupport.PrepareUpload
0 0% 97.74% 243.54MB 4.87% google.golang.org/api/storage/v1.(*ObjectsInsertCall).Do
0 0% 97.74% 1472MB 29.44% google.golang.org/api/storage/v1.(*ObjectsInsertCall).Media
0 0% 97.74% 341.04MB 6.82% google.golang.org/api/storage/v1.(*ObjectsListCall).Do
0 0% 97.74% 136.01MB 2.72% path/filepath.Join (inline)
0 0% 97.74% 136.01MB 2.72% path/filepath.join
0 0% 97.74% 136.01MB 2.72% strings.(*Builder).Grow (inline)
0 0% 97.74% 136.01MB 2.72% strings.Join
must change profile007.txt to profile007.zipprofile007.txt (22.0 KB)
