Files are from 16Kb and up to 500Mb each. Average - 2-3Mb each.
Usage - both reading and writing. Usually - reading. Writing in most cases - only one time.
I'll try to produce a memory profile tomorrow. I cannot unmount the storage now.
system is running for 2 hours.
According to the htop, rclone took about 8Gb of RAM.
1533383 root 20 0 8733M 7970M 31492 S 15.3 24.8 33:15.72 rclone mount --log-f...
rclone mount --log-file=/var/log/dcmsys/rclone.log --log-level ERROR --config=/etc/rclone.conf --allow-other --default-permissions --max-read-ahead=4M --vfs-cache-mode=writes --vfs-cache-max-size=1G --azureblob-memory-pool-flush-time=1m0s --buffer-size=0 --azureblob-memory-pool-use-mmap --use-mmap --dir-cache-time=2h --timeout=1h --cache-dir=/var/dcmsys/cache --uid=1100 --gid=1100 --cache-chunk-no-memory --rc --size-only --allow-non-empty blob:studiesh /var/dcmsys/storages/GCP
I have file pprof.rclone.alloc_objects.alloc_space.inuse_objects.inuse_space.026.pb.gz but not sure how to share it here.
Fetching profile over HTTP from http://localhost:5572/debug/pprof/heap
Saved profile in /root/pprof/pprof.rclone.alloc_objects.alloc_space.inuse_objects.inuse_space.026.pb.gz
File: rclone
Type: inuse_space
Time: Jan 8, 2021 at 11:10am (EST)
Showing nodes accounting for 3758.28MB, 96.70% of 3886.48MB total
Dropped 131 nodes (cum <= 19.43MB)
flat flat% sum% cum cum%
3605.21MB 92.76% 92.76% 3605.21MB 92.76% github.com/Azure/azure-storage-blob-go/azblob.copyFromReader.func1
40.51MB 1.04% 93.81% 40.51MB 1.04% github.com/Azure/azure-storage-blob-go/azblob.BlobGetPropertiesResponse.NewMetadata
25MB 0.64% 94.45% 25MB 0.64% github.com/rclone/rclone/vfs.newFile
24.01MB 0.62% 95.07% 29.01MB 0.75% github.com/rclone/rclone/vfs/vfscache.newItem
20MB 0.51% 95.58% 20MB 0.51% strings.(*Builder).grow (inline)
14MB 0.36% 95.94% 23.71MB 0.61% context.WithDeadline
12.50MB 0.32% 96.26% 3327.12MB 85.61% github.com/rclone/rclone/backend/azureblob.(*Fs).Put
8MB 0.21% 96.47% 26.79MB 0.69% github.com/rclone/rclone/vfs.newRWFileHandle
4.04MB 0.1% 96.57% 21.04MB 0.54% github.com/rclone/rclone/vfs.(*Dir)._readDirFromEntries
2MB 0.051% 96.62% 100.30MB 2.58% github.com/rclone/rclone/cmd/mount.(*Dir).Create
1.50MB 0.039% 96.66% 26.22MB 0.67% github.com/Azure/azure-storage-blob-go/azblob.(*SharedKeyCredential).New.func1
1MB 0.026% 96.69% 30.01MB 0.77% github.com/rclone/rclone/vfs/vfscache.(*Cache)._get
0.50MB 0.013% 96.70% 24.53MB 0.63% github.com/rclone/rclone/cmd/mount.(*Dir).Lookup
0 0% 96.70% 137.96MB 3.55% bazil.org/fuse/fs.(*Server).Serve.func1
0 0% 96.70% 137.96MB 3.55% bazil.org/fuse/fs.(*Server).handleRequest
0 0% 96.70% 137.96MB 3.55% bazil.org/fuse/fs.(*Server).serve
0 0% 96.70% 23.71MB 0.61% context.WithTimeout
0 0% 96.70% 50.43MB 1.30% github.com/Azure/azure-pipeline-go/pipeline.(*pipeline).Do
0 0% 96.70% 50.43MB 1.30% github.com/Azure/azure-pipeline-go/pipeline.PolicyFunc.Do
0 0% 96.70% 3606.21MB 92.79% github.com/Azure/azure-storage-blob-go/azblob.(*copier).sendChunk
0 0% 96.70% 28.50MB 0.73% github.com/Azure/azure-storage-blob-go/azblob.BlobURL.GetProperties
0 0% 96.70% 50.43MB 1.30% github.com/Azure/azure-storage-blob-go/azblob.NewRetryPolicyFactory.func1.1
0 0% 96.70% 50.43MB 1.30% github.com/Azure/azure-storage-blob-go/azblob.NewTelemetryPolicyFactory.func1.1
0 0% 96.70% 50.43MB 1.30% github.com/Azure/azure-storage-blob-go/azblob.NewUniqueRequestIDPolicyFactory.func1.1
0 0% 96.70% 3611.92MB 92.94% github.com/Azure/azure-storage-blob-go/azblob.UploadStreamToBlockBlob
0 0% 96.70% 28.50MB 0.73% github.com/Azure/azure-storage-blob-go/azblob.blobClient.GetProperties
0 0% 96.70% 3611.92MB 92.94% github.com/Azure/azure-storage-blob-go/azblob.copyFromReader
0 0% 96.70% 24.71MB 0.64% github.com/Azure/azure-storage-blob-go/azblob.responderPolicy.Do
0 0% 96.70% 20MB 0.51% github.com/rclone/rclone/backend/azureblob.(*Fs).list
0 0% 96.70% 3686.24MB 94.85% github.com/rclone/rclone/backend/azureblob.(*Object).Update
0 0% 96.70% 3611.92MB 92.94% github.com/rclone/rclone/backend/azureblob.(*Object).Update.func1
0 0% 96.70% 46.01MB 1.18% github.com/rclone/rclone/backend/azureblob.(*Object).decodeMetaDataFromPropertiesResponse
0 0% 96.70% 74.51MB 1.92% github.com/rclone/rclone/backend/azureblob.(*Object).readMetaData
0 0% 96.70% 28.50MB 0.73% github.com/rclone/rclone/backend/azureblob.(*Object).readMetaData.func1
0 0% 96.70% 3652.22MB 93.97% github.com/rclone/rclone/fs.pacerInvoker
0 0% 96.70% 3702.75MB 95.27% github.com/rclone/rclone/fs/operations.Copy
0 0% 96.70% 41MB 1.06% github.com/rclone/rclone/lib/pacer.(*Pacer).Call
0 0% 96.70% 3611.21MB 92.92% github.com/rclone/rclone/lib/pacer.(*Pacer).CallNoRetry (inline)
0 0% 96.70% 3652.22MB 93.97% github.com/rclone/rclone/lib/pacer.(*Pacer).call
0 0% 96.70% 21.50MB 0.55% github.com/rclone/rclone/vfs.(*Dir).Create
0 0% 96.70% 24.03MB 0.62% github.com/rclone/rclone/vfs.(*Dir).Stat
0 0% 96.70% 29.54MB 0.76% github.com/rclone/rclone/vfs.(*Dir)._readDir
0 0% 96.70% 24.03MB 0.62% github.com/rclone/rclone/vfs.(*Dir).stat
0 0% 96.70% 76.80MB 1.98% github.com/rclone/rclone/vfs.(*File).Open
0 0% 96.70% 20MB 0.51% github.com/rclone/rclone/vfs.(*File).Path
0 0% 96.70% 26.79MB 0.69% github.com/rclone/rclone/vfs.(*File).openRW
0 0% 96.70% 30.01MB 0.77% github.com/rclone/rclone/vfs/vfscache.(*Cache).Exists
0 0% 96.70% 30.01MB 0.77% github.com/rclone/rclone/vfs/vfscache.(*Cache).get
0 0% 96.70% 3694.24MB 95.05% github.com/rclone/rclone/vfs/vfscache.(*Item).Close.func2
0 0% 96.70% 3694.24MB 95.05% github.com/rclone/rclone/vfs/vfscache.(*Item)._store
0 0% 96.70% 3694.24MB 95.05% github.com/rclone/rclone/vfs/vfscache.(*Item).store
0 0% 96.70% 3694.24MB 95.05% github.com/rclone/rclone/vfs/vfscache/writeback.(*WriteBack).upload
0 0% 96.70% 20MB 0.51% path.Join
0 0% 96.70% 20MB 0.51% strings.(*Builder).Grow (inline)
0 0% 96.70% 20MB 0.51% strings.Join
0 0% 96.70% 3613.04MB 92.96% sync.(*Pool).Get
Best,
Raymond
Hmm, the leak is in the Azure blob SDK.
I have a feeling that this is fixed in the SDK version in use in the beta.
Was that from the beta?
Maybe I forgot to merge something.
I used rclone version:
rclone v1.54.0-beta.5040.71edc75ca
I dug into this again and remembered I made an issue about this on the azure blob SDK
@Raimondas Until the bug @ncw mentioned in the Azure blob SDK is fixed I suggest you stick with rclone 1.52.3, that version works fine with low memory usage (can't say for sure about CPU but I think the high usage might be related to garbage collection).
See my original post about the problem here, I think it is the same as yours: Rclone 1.53.3 memory (leak?) problem
Hi Daniel,
You saved the day! Thanks a lot for your advice!
I installed rclone v1.52.3 and after 1h45m I see rclone has eaten only 450Mb. That's success!
Saved profile in /root/pprof/pprof.rclone.alloc_objects.alloc_space.inuse_objects.inuse_space.035.pb.gz
File: rclone
Type: inuse_space
Time: Jan 9, 2021 at 12:33pm (EST)
Showing nodes accounting for 138.46MB, 98.19% of 141.01MB total
Dropped 25 nodes (cum <= 0.71MB)
flat flat% sum% cum cum%
20.50MB 14.54% 14.54% 20.50MB 14.54% strings.(*Builder).grow (inline)
18MB 12.77% 27.31% 18MB 12.77% github.com/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.BlobGetPropertiesResponse.NewMetadata
13.50MB 9.58% 36.88% 13.50MB 9.58% github.com/rclone/rclone/vfs.newFile
12MB 8.51% 45.40% 12MB 8.51% github.com/rclone/rclone/vfs.newDir
8.81MB 6.24% 51.64% 8.81MB 6.24% github.com/rclone/rclone/vendor/bazil.org/fuse/fs.(*Server).saveNode
7MB 4.96% 56.60% 46.01MB 32.63% github.com/rclone/rclone/backend/azureblob.(*Fs).Put
6MB 4.26% 60.86% 6MB 4.26% time.AfterFunc
6MB 4.26% 65.12% 6MB 4.26% net/textproto.(*Reader).ReadMIMEHeader
5.50MB 3.90% 69.02% 11.50MB 8.16% context.WithDeadline
5.26MB 3.73% 72.74% 5.26MB 3.73% github.com/rclone/rclone/vfs.(*Dir).addObject
4.54MB 3.22% 75.96% 17.54MB 12.44% github.com/rclone/rclone/vfs.(*Dir)._readDirFromEntries
4.50MB 3.19% 79.16% 4.50MB 3.19% os.newFile
4MB 2.84% 81.99% 4MB 2.84% github.com/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.uploadStream.func1
4MB 2.84% 84.83% 4MB 2.84% context.(*cancelCtx).Done
4MB 2.84% 87.67% 4MB 2.84% encoding/xml.copyValue
3.50MB 2.48% 90.15% 27.76MB 19.69% github.com/rclone/rclone/vfs.newRWFileHandle
3MB 2.13% 92.28% 3MB 2.13% github.com/rclone/rclone/fs.NewDir (inline)
2.50MB 1.77% 94.05% 3.64MB 2.58% github.com/rclone/rclone/vendor/bazil.org/fuse.(*Conn).ReadRequest
1.50MB 1.06% 95.11% 1.50MB 1.06% github.com/rclone/rclone/vendor/github.com/aws/aws-sdk-go/aws/endpoints.init
1.50MB 1.06% 96.18% 1.50MB 1.06% github.com/rclone/rclone/vfs.(*File).addWriter
1.20MB 0.85% 97.03% 1.20MB 0.85% time.startTimer
1.14MB 0.81% 97.84% 1.14MB 0.81% github.com/rclone/rclone/vendor/bazil.org/fuse.allocMessage
0.50MB 0.35% 98.19% 1.10MB 0.78% time.NewTimer
0 0% 98.19% 11.50MB 8.16% context.WithTimeout
0 0% 98.19% 1.05MB 0.74% crypto/tls.(*Conn).Handshake
0 0% 98.19% 1.05MB 0.74% crypto/tls.(*Conn).clientHandshake
0 0% 98.19% 1.05MB 0.74% crypto/tls.(*clientHandshakeState).doFullHandshake
0 0% 98.19% 1.05MB 0.74% crypto/tls.(*clientHandshakeState).handshake
0 0% 98.19% 4MB 2.84% encoding/xml.(*Decoder).Decode (inline)
0 0% 98.19% 4MB 2.84% encoding/xml.(*Decoder).DecodeElement
0 0% 98.19% 4MB 2.84% encoding/xml.(*Decoder).unmarshal
0 0% 98.19% 4MB 2.84% encoding/xml.(*Decoder).unmarshalPath
0 0% 98.19% 4MB 2.84% encoding/xml.Unmarshal
0 0% 98.19% 7MB 4.96% github.com/rclone/rclone/backend/azureblob.(*Fs).List
0 0% 98.19% 3MB 2.13% github.com/rclone/rclone/backend/azureblob.(*Fs).itemToDirEntry
0 0% 98.19% 7MB 4.96% github.com/rclone/rclone/backend/azureblob.(*Fs).list
0 0% 98.19% 4MB 2.84% github.com/rclone/rclone/backend/azureblob.(*Fs).list.func1
0 0% 98.19% 7MB 4.96% github.com/rclone/rclone/backend/azureblob.(*Fs).listDir
0 0% 98.19% 3MB 2.13% github.com/rclone/rclone/backend/azureblob.(*Fs).listDir.func1
0 0% 98.19% 39.01MB 27.66% github.com/rclone/rclone/backend/azureblob.(*Object).Update
0 0% 98.19% 5MB 3.55% github.com/rclone/rclone/backend/azureblob.(*Object).Update.func1
0 0% 98.19% 18.50MB 13.12% github.com/rclone/rclone/backend/azureblob.(*Object).decodeMetaDataFromPropertiesResponse
0 0% 98.19% 34.01MB 24.12% github.com/rclone/rclone/backend/azureblob.(*Object).readMetaData
0 0% 98.19% 16.10MB 11.42% github.com/rclone/rclone/backend/azureblob.(*Object).readMetaData.func1
0 0% 98.19% 5.10MB 3.62% github.com/rclone/rclone/backend/azureblob.httpClientFactory.func1.1
0 0% 98.19% 40.26MB 28.55% github.com/rclone/rclone/cmd/mount.(*Dir).Create
0 0% 98.19% 19.54MB 13.86% github.com/rclone/rclone/cmd/mount.(*Dir).Lookup
0 0% 98.19% 1MB 0.71% github.com/rclone/rclone/cmd/mount.(*Dir).ReadDirAll
0 0% 98.19% 49.51MB 35.11% github.com/rclone/rclone/cmd/mount.(*FileHandle).Flush
0 0% 98.19% 3.64MB 2.58% github.com/rclone/rclone/cmd/mount.mount.func1
0 0% 98.19% 25.10MB 17.80% github.com/rclone/rclone/fs.pacerInvoker
0 0% 98.19% 5.10MB 3.62% github.com/rclone/rclone/fs/fshttp.(*Transport).RoundTrip
0 0% 98.19% 3MB 2.13% github.com/rclone/rclone/fs/list.DirSorted
0 0% 98.19% 46.01MB 32.63% github.com/rclone/rclone/fs/operations.Copy
0 0% 98.19% 20.10MB 14.26% github.com/rclone/rclone/lib/pacer.(*Pacer).Call
0 0% 98.19% 5MB 3.55% github.com/rclone/rclone/lib/pacer.(*Pacer).CallNoRetry (inline)
0 0% 98.19% 25.10MB 17.80% github.com/rclone/rclone/lib/pacer.(*Pacer).call
0 0% 98.19% 1.14MB 0.81% github.com/rclone/rclone/vendor/bazil.org/fuse.getMessage (inline)
0 0% 98.19% 3.64MB 2.58% github.com/rclone/rclone/vendor/bazil.org/fuse/fs.(*Server).Serve
0 0% 98.19% 119.12MB 84.48% github.com/rclone/rclone/vendor/bazil.org/fuse/fs.(*Server).Serve.func1
0 0% 98.19% 119.12MB 84.48% github.com/rclone/rclone/vendor/bazil.org/fuse/fs.(*Server).handleRequest
0 0% 98.19% 8.81MB 6.24% github.com/rclone/rclone/vendor/bazil.org/fuse/fs.(*Server).saveLookup
0 0% 98.19% 119.12MB 84.48% github.com/rclone/rclone/vendor/bazil.org/fuse/fs.(*Server).serve
0 0% 98.19% 20.60MB 14.61% github.com/rclone/rclone/vendor/github.com/Azure/azure-pipeline-go/pipeline.(*pipeline).Do
0 0% 98.19% 20.60MB 14.61% github.com/rclone/rclone/vendor/github.com/Azure/azure-pipeline-go/pipeline.PolicyFunc.Do
0 0% 98.19% 9.10MB 6.45% github.com/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.(*SharedKeyCredential).New.func1
0 0% 98.19% 16.10MB 11.42% github.com/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.BlobURL.GetProperties
0 0% 98.19% 4MB 2.84% github.com/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.ContainerURL.ListBlobsHierarchySegment
0 0% 98.19% 5.10MB 3.62% github.com/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.NewRequestLogPolicyFactory.func1.1
0 0% 98.19% 20.60MB 14.61% github.com/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.NewRetryPolicyFactory.func1.1
0 0% 98.19% 20.60MB 14.61% github.com/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.NewTelemetryPolicyFactory.func1.1
0 0% 98.19% 20.60MB 14.61% github.com/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.NewUniqueRequestIDPolicyFactory.func1.1
0 0% 98.19% 5MB 3.55% github.com/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.UploadStreamToBlockBlob
0 0% 98.19% 16.10MB 11.42% github.com/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.blobClient.GetProperties
0 0% 98.19% 4MB 2.84% github.com/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.containerClient.ListBlobHierarchySegment
0 0% 98.19% 4MB 2.84% github.com/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.containerClient.listBlobHierarchySegmentResponder
0 0% 98.19% 9.10MB 6.45% github.com/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.responderPolicy.Do
0 0% 98.19% 5MB 3.55% github.com/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.uploadStream
0 0% 98.19% 12.50MB 8.87% github.com/rclone/rclone/vfs.(*Dir).Create
0 0% 98.19% 1MB 0.71% github.com/rclone/rclone/vfs.(*Dir).ReadDirAll
0 0% 98.19% 19.54MB 13.86% github.com/rclone/rclone/vfs.(*Dir).Stat
0 0% 98.19% 20.54MB 14.57% github.com/rclone/rclone/vfs.(*Dir)._readDir
0 0% 98.19% 19.54MB 13.86% github.com/rclone/rclone/vfs.(*Dir).stat
0 0% 98.19% 27.76MB 19.69% github.com/rclone/rclone/vfs.(*File).Open
0 0% 98.19% 9MB 6.38% github.com/rclone/rclone/vfs.(*File).Path
0 0% 98.19% 27.76MB 19.69% github.com/rclone/rclone/vfs.(*File).openRW
0 0% 98.19% 11.50MB 8.16% github.com/rclone/rclone/vfs.(*File).osPath
0 0% 98.19% 49.51MB 35.11% github.com/rclone/rclone/vfs.(*RWFileHandle).Flush
0 0% 98.19% 49.51MB 35.11% github.com/rclone/rclone/vfs.(*RWFileHandle).flushWrites
0 0% 98.19% 21.26MB 15.08% github.com/rclone/rclone/vfs.(*RWFileHandle).openPending
0 0% 98.19% 42.01MB 29.79% github.com/rclone/rclone/vfs/vfscache.(*Cache).Store
0 0% 98.19% 11.50MB 8.16% github.com/rclone/rclone/vfs/vfscache.(*Cache).ToOSPath (inline)
0 0% 98.19% 42.01MB 29.79% github.com/rclone/rclone/vfs/vfscache.copyObj
0 0% 98.19% 5.10MB 3.62% net/http.(*Client).Do (inline)
0 0% 98.19% 5.10MB 3.62% net/http.(*Client).do
0 0% 98.19% 5.10MB 3.62% net/http.(*Client).send
0 0% 98.19% 5.10MB 3.62% net/http.(*Transport).RoundTrip (inline)
0 0% 98.19% 5.10MB 3.62% net/http.(*Transport).roundTrip
0 0% 98.19% 1.05MB 0.74% net/http.(*persistConn).addTLS.func2
0 0% 98.19% 6MB 4.26% net/http.(*persistConn).readLoop
0 0% 98.19% 6MB 4.26% net/http.(*persistConn).readResponse
0 0% 98.19% 1.10MB 0.78% net/http.(*persistConn).roundTrip
0 0% 98.19% 6MB 4.26% net/http.ReadResponse
0 0% 98.19% 5.10MB 3.62% net/http.send
0 0% 98.19% 4.50MB 3.19% os.OpenFile
0 0% 98.19% 4.50MB 3.19% os.openFileNolog
0 0% 98.19% 9MB 6.38% path.Join
0 0% 98.19% 11.50MB 8.16% path/filepath.Join (inline)
0 0% 98.19% 11.50MB 8.16% path/filepath.join
0 0% 98.19% 1.50MB 1.06% runtime.doInit
0 0% 98.19% 1.50MB 1.06% runtime.main
0 0% 98.19% 20.50MB 14.54% strings.(*Builder).Grow (inline)
0 0% 98.19% 20.50MB 14.54% strings.Join
0 0% 98.19% 1.05MB 0.74% sync.(*Once).Do (inline)
0 0% 98.19% 1.05MB 0.74% sync.(*Once).doSlow
0 0% 98.19% 1.14MB 0.81% sync.(*Pool).Get
Best,
Raymond
update.
rclone v1.52.3
rclone mount --log-file=/var/log/dcmsys/rclone.log --log-level ERROR --config=/etc/rclone.conf --allow-other --default-permissions --max-read-ahead=4M --vfs-cache-mode=writes --vfs-cache-max-size=1G --azureblob-memory-pool-flush-time=1m0s --buffer-size=0 --azureblob-memory-pool-use-mmap --use-mmap --dir-cache-time=2h --timeout=1h --cache-dir=/var/dcmsys/cache --uid=1100 --gid=1100 --cache-chunk-no-memory --rc --size-only --allow-non-empty blob:studiesh /var/dcmsys/storages/GCP
after 2 days of running, htop shows that rclone uses about 9Gb or ram:
1860391 root 20 0 9.8G 9210M 10612 S 0.0 28.6 18h41:38 rclone mount --log-file=/var/log/dcmsys/rclone.log --log-level ERROR --config=/etc/rcl
Here is the memory profile:
> Fetching profile over HTTP from localhost:5572/debug/pprof/heap
> Saved profile in /root/pprof/pprof.rclone.alloc_objects.alloc_space.inuse_objects.inuse_space.037.pb.gz
> File: rclone
> Type: inuse_space
> Time: Jan 11, 2021 at 9:10am (EST)
> Showing nodes accounting for 4060.02MB, 97.81% of 4151MB total
> Dropped 99 nodes (cum <= 20.75MB)
> flat flat% sum% cum cum%
> 712.17MB 17.16% 17.16% 712.17MB 17.16% github.c/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.BlobGetPropertiesResponse.NewMetadata
> 674.10MB 16.24% 33.40% 674.10MB 16.24% strings.(*Builder).grow (inline)
> 521.61MB 12.57% 45.96% 521.61MB 12.57% github.c/rclone/rclone/vfs.newFile
> 270.02MB 6.50% 52.47% 487.07MB 11.73% context.WithDeadline
> 258.01MB 6.22% 58.68% 258.01MB 6.22% net/textproto.(*Reader).ReadMIMEHeader
> 245.52MB 5.91% 64.60% 245.52MB 5.91% context.(*cancelCtx).Done
> 242.03MB 5.83% 70.43% 1852.36MB 44.62% github.c/rclone/rclone/backend/azureblob.(*Fs).Put
> 232.52MB 5.60% 76.03% 232.52MB 5.60% os.newFile
> 180.51MB 4.35% 80.38% 198.55MB 4.78% time.AfterFunc
> 161.33MB 3.89% 84.26% 161.33MB 3.89% github.c/rclone/rclone/vfs.(*Dir).addObject
> 146.51MB 3.53% 87.79% 964.89MB 23.24% github.c/rclone/rclone/vfs.newRWFileHandle
> 141.01MB 3.40% 91.19% 153.53MB 3.70% github.c/rclone/rclone/vendor/bazil.org/fuse.(*Conn).ReadRequest
> 128.04MB 3.08% 94.28% 128.04MB 3.08% github.c/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.uploadStream.func1
> 64MB 1.54% 95.82% 64MB 1.54% encoding/base64.(*Encoding).EncodeToString
> 40.50MB 0.98% 96.79% 40.50MB 0.98% github.c/rclone/rclone/vfs.(*File).addWriter
> 23.58MB 0.57% 97.36% 23.58MB 0.57% time.startTimer
> 11.50MB 0.28% 97.64% 1496.51MB 36.05% github.c/rclone/rclone/cmd/mount.(*Dir).Create
> 7.06MB 0.17% 97.81% 25.06MB 0.6% github.c/rclone/rclone/vfs.(*Dir)._readDirFromEntries
> 0 0% 97.81% 487.07MB 11.73% context.WithTimeout
> 0 0% 97.81% 21MB 0.51% encoding/xml.(*Decoder).Decode (inline)
> 0 0% 97.81% 21MB 0.51% encoding/xml.(*Decoder).DecodeElement
> 0 0% 97.81% 21MB 0.51% encoding/xml.(*Decoder).unmarshal
> 0 0% 97.81% 21MB 0.51% encoding/xml.(*Decoder).unmarshalPath
> 0 0% 97.81% 21MB 0.51% encoding/xml.Unmarshal
> 0 0% 97.81% 27MB 0.65% github.c/rclone/rclone/backend/azureblob.(*Fs).List
> 0 0% 97.81% 37MB 0.89% github.c/rclone/rclone/backend/azureblob.(*Fs).list
> 0 0% 97.81% 27.50MB 0.66% github.c/rclone/rclone/backend/azureblob.(*Fs).list.func1
> 0 0% 97.81% 27MB 0.65% github.c/rclone/rclone/backend/azureblob.(*Fs).listDir
> 0 0% 97.81% 1623.84MB 39.12% github.c/rclone/rclone/backend/azureblob.(*Object).Update
> 0 0% 97.81% 224.08MB 5.40% github.c/rclone/rclone/backend/azureblob.(*Object).Update.func1
> 0 0% 97.81% 774.68MB 18.66% github.c/rclone/rclone/backend/azureblob.(*Object).decodeMetaDataFromPropertiesResponse
> 0 0% 97.81% 1399.23MB 33.71% github.c/rclone/rclone/backend/azureblob.(*Object).readMetaData
> 0 0% 97.81% 624.05MB 15.03% github.c/rclone/rclone/backend/azureblob.(*Object).readMetaData.func1
> 0 0% 97.81% 228.52MB 5.51% github.c/rclone/rclone/backend/azureblob.httpClientFactory.func1.1
> 0 0% 97.81% 35.56MB 0.86% github.c/rclone/rclone/cmd/mount.(*Dir).Lookup
> 0 0% 97.81% 1923.88MB 46.35% github.c/rclone/rclone/cmd/mount.(*FileHandle).Flush
> 0 0% 97.81% 153.53MB 3.70% github.c/rclone/rclone/cmd/mount.mount.func1
> 0 0% 97.81% 876.14MB 21.11% github.c/rclone/rclone/fs.pacerInvoker
> 0 0% 97.81% 228.52MB 5.51% github.c/rclone/rclone/fs/fshttp.(*Transport).RoundTrip
> 0 0% 97.81% 1853.87MB 44.66% github.c/rclone/rclone/fs/operations.Copy
> 0 0% 97.81% 651.56MB 15.70% github.c/rclone/rclone/lib/pacer.(*Pacer).Call
> 0 0% 97.81% 223.58MB 5.39% github.c/rclone/rclone/lib/pacer.(*Pacer).CallNoRetry (inline)
> 0 0% 97.81% 876.14MB 21.11% github.c/rclone/rclone/lib/pacer.(*Pacer).call
> 0 0% 97.81% 153.53MB 3.70% github.c/rclone/rclone/vendor/bazil.org/fuse/fs.(*Server).Serve
> 0 0% 97.81% 3459.25MB 83.34% github.c/rclone/rclone/vendor/bazil.org/fuse/fs.(*Server).Serve.func1
> 0 0% 97.81% 3477.28MB 83.77% github.c/rclone/rclone/vendor/bazil.org/fuse/fs.(*Server).handleRequest
> 0 0% 97.81% 3477.28MB 83.77% github.c/rclone/rclone/vendor/bazil.org/fuse/fs.(*Server).serve
> 0 0% 97.81% 737.59MB 17.77% github.c/rclone/rclone/vendor/github.com/Azure/azure-pipeline-go/pipeline.(*pipeline).Do
> 0 0% 97.81% 737.59MB 17.77% github.c/rclone/rclone/vendor/github.c/Azure/azure-pipeline-go/pipeline.PolicyFunc.Do
> 0 0% 97.81% 250.02MB 6.02% github.c/rclone/rclone/vendor/github.c/Azure/azure-storage-blob-go/azblob.(*SharedKeyCredential).New.func1
> 0 0% 97.81% 83.54MB 2.01% github.c/rclone/rclone/vendor/github.c/Azure/azure-storage-blob-go/azblob.(*uploadStreamToBlockBlobOptions).end
> 0 0% 97.81% 624.05MB 15.03% github.c/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.BlobURL.GetProperties
> 0 0% 97.81% 83.54MB 2.01% github.c/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.BlockBlobURL.Upload
> 0 0% 97.81% 27.50MB 0.66% github.c/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.ContainerURL.ListBlobsHierarchySegment
> 0 0% 97.81% 228.52MB 5.51% github.c/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.NewRequestLogPolicyFactory.func1.1
> 0 0% 97.81% 737.59MB 17.77% github.c/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.NewRetryPolicyFactory.func1.1
> 0 0% 97.81% 737.59MB 17.77% github.c/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.NewTelemetryPolicyFactory.func1.1
> 0 0% 97.81% 737.59MB 17.77% github./rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.NewUniqueRequestIDPolicyFactory.func1.1
> 0 0% 97.81% 224.08MB 5.40% github./rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.UploadStreamToBlockBlob
> 0 0% 97.81% 624.05MB 15.03% github./rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.blobClient.GetProperties
> 0 0% 97.81% 83.54MB 2.01% github./rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.blockBlobClient.Upload
> 0 0% 97.81% 27.50MB 0.66% github./rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.containerClient.ListBlobHierarchySegment
> 0 0% 97.81% 21MB 0.51% github./rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.containerClient.listBlobHierarchySegmentResponder
> 0 0% 97.81% 249.52MB 6.01% github./rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.responderPolicy.Do
> 0 0% 97.81% 224.08MB 5.40% github./rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.uploadStream
> 0 0% 97.81% 520.11MB 12.53% github./rclone/rclone/vfs.(*Dir).Create
> 0 0% 97.81% 35.56MB 0.86% github./rclone/rclone/vfs.(*Dir).Stat
> 0 0% 97.81% 39.56MB 0.95% github.c/rclone/rclone/vfs.(*Dir)._readDir
> 0 0% 97.81% 35.56MB 0.86% github./rclone/rclone/vfs.(*Dir).stat
> 0 0% 97.81% 964.89MB 23.24% github./rclone/rclone/vfs.(*File).Open
> 0 0% 97.81% 282.03MB 6.79% github./rclone/rclone/vfs.(*File).Path
> 0 0% 97.81% 964.89MB 23.24% github./rclone/rclone/vfs.(*File).openRW
> 0 0% 97.81% 389.56MB 9.38% github/rclone/rclone/vfs.(*File).osPath
> 0 0% 97.81% 1923.88MB 46.35% github./rclone/rclone/vfs.(*RWFileHandle).Flush
> 0 0% 97.81% 1923.88MB 46.35% github.c/rclone/rclone/vfs.(*RWFileHandle).flushWrites
> 0 0% 97.81% 781.89MB 18.84% github/rclone/rclone/vfs.(*RWFileHandle).openPending
> 0 0% 97.81% 1641.85MB 39.55% github.c/rclone/rclone/vfs/vfscache.(*Cache).Store
> 0 0% 97.81% 389.56MB 9.38% github.c/rclone/rclone/vfs/vfscache.(*Cache).ToOSPath (inline)
> 0 0% 97.81% 1641.85MB 39.55% github.c/rclone/rclone/vfs/vfscache.copyObj
> 0 0% 97.81% 228.52MB 5.51% net/http.(*Client).Do (inline)
> 0 0% 97.81% 228.52MB 5.51% net/http.(*Client).do
> 0 0% 97.81% 228.52MB 5.51% net/http.(*Client).send
> 0 0% 97.81% 228.52MB 5.51% net/http.(*Transport).RoundTrip (inline)
> 0 0% 97.81% 228.52MB 5.51% net/http.(*Transport).roundTrip
> 0 0% 97.81% 258.01MB 6.22% net/http.(*persistConn).readLoop
> 0 0% 97.81% 258.01MB 6.22% net/http.(*persistConn).readResponse
> 0 0% 97.81% 258.01MB 6.22% net/http.ReadResponse
> 0 0% 97.81% 228.52MB 5.51% net/http.send
> 0 0% 97.81% 232.52MB 5.60% os.OpenFile
> 0 0% 97.81% 232.52MB 5.60% os.openFileNolog
> 0 0% 97.81% 284.03MB 6.84% path.Join
> 0 0% 97.81% 389.56MB 9.38% path/filepath.Join (inline)
> 0 0% 97.81% 389.56MB 9.38% path/filepath.join
> 0 0% 97.81% 674.10MB 16.24% strings.(*Builder).Grow (inline)
> 0 0% 97.81% 673.60MB 16.23% strings.Join
Best,
Raymond
Would you mind trying this beta or later?
This has had the large file upload code reworked to use the facilities from the SDK entirely. I think this may help, but I'm not 100% sure!
Uptime 3h:
rclone v1.54.0-beta.5060.e0b5a13a1
- os/arch: linux/amd64
- go version: go1.15.6
htop:
2703199 root 20 0 10.6G 9745M 31376 S 24.6 30.3 40:39.95 rclone mount --log-file=/var/log/dcmsys/rclone.log --log-level ERROR --config=/etc/rclone.conf --allow-other --default-permissions --max-read-ahead=4M --vfs-cache-mode=writes --vfs-cache-max-size=1G --azureblob-memory-pool-f
Saved profile in /root/pprof/pprof.rclone.alloc_objects.alloc_space.inuse_objects.inuse_space.041.pb.gz
File: rclone
Type: inuse_space
Time: Jan 13, 2021 at 4:15am (EST)
Showing nodes accounting for 4827.99MB, 96.60% of 4997.81MB total
Dropped 147 nodes (cum <= 24.99MB)
flat flat% sum% cum cum%
4601.54MB 92.07% 92.07% 4601.54MB 92.07% github.com/Azure/azure-storage-blob-go/azblob.copyFromReader.func1
58.01MB 1.16% 93.23% 58.01MB 1.16% github.com/Azure/azure-storage-blob-go/azblob.BlobGetPropertiesResponse.NewMetadata
36.01MB 0.72% 93.95% 46.01MB 0.92% github.com/rclone/rclone/vfs/vfscache.newItem
34.01MB 0.68% 94.63% 34.01MB 0.68% github.com/rclone/rclone/vfs.newFile
27.33MB 0.55% 95.18% 27.33MB 0.55% bazil.org/fuse.allocMessage
22.50MB 0.45% 95.63% 31.77MB 0.64% context.WithDeadline
15.50MB 0.31% 95.94% 4735.57MB 94.75% github.com/rclone/rclone/backend/azureblob.(*Fs).Put
13.50MB 0.27% 96.21% 26.74MB 0.53% github.com/rclone/rclone/vfs.newRWFileHandle
9.50MB 0.19% 96.40% 36.83MB 0.74% bazil.org/fuse.(*Conn).ReadRequest
5.09MB 0.1% 96.50% 27.09MB 0.54% github.com/rclone/rclone/vfs.(*Dir)._readDirFromEntries
2MB 0.04% 96.54% 38.55MB 0.77% github.com/Azure/azure-storage-blob-go/azblob.(*SharedKeyCredential).New.func1
1.50MB 0.03% 96.57% 129.75MB 2.60% github.com/rclone/rclone/cmd/mount.(*Dir).Create
0.50MB 0.01% 96.58% 4610.32MB 92.25% github.com/Azure/azure-storage-blob-go/azblob.copyFromReader
0.50MB 0.01% 96.59% 4610.82MB 92.26% github.com/Azure/azure-storage-blob-go/azblob.UploadStreamToBlockBlob
0.50MB 0.01% 96.60% 70.82MB 1.42% github.com/Azure/azure-storage-blob-go/azblob.NewRetryPolicyFactory.func1.1
0 0% 96.60% 27.33MB 0.55% bazil.org/fuse.getMessage (inline)
0 0% 96.60% 36.83MB 0.74% bazil.org/fuse/fs.(*Server).Serve
0 0% 96.60% 176.84MB 3.54% bazil.org/fuse/fs.(*Server).Serve.func1
0 0% 96.60% 176.84MB 3.54% bazil.org/fuse/fs.(*Server).handleRequest
0 0% 96.60% 176.84MB 3.54% bazil.org/fuse/fs.(*Server).serve
0 0% 96.60% 31.77MB 0.64% context.WithTimeout
0 0% 96.60% 70.82MB 1.42% github.com/Azure/azure-pipeline-go/pipeline.(*pipeline).Do
0 0% 96.60% 70.82MB 1.42% github.com/Azure/azure-pipeline-go/pipeline.PolicyFunc.Do
0 0% 96.60% 4602.04MB 92.08% github.com/Azure/azure-storage-blob-go/azblob.(*copier).sendChunk
0 0% 96.60% 47.50MB 0.95% github.com/Azure/azure-storage-blob-go/azblob.BlobURL.GetProperties
0 0% 96.60% 25.54MB 0.51% github.com/Azure/azure-storage-blob-go/azblob.NewRequestLogPolicyFactory.func1.1
0 0% 96.60% 70.82MB 1.42% github.com/Azure/azure-storage-blob-go/azblob.NewTelemetryPolicyFactory.func1.1
0 0% 96.60% 70.82MB 1.42% github.com/Azure/azure-storage-blob-go/azblob.NewUniqueRequestIDPolicyFactory.func1.1
0 0% 96.60% 47.50MB 0.95% github.com/Azure/azure-storage-blob-go/azblob.blobClient.GetProperties
0 0% 96.60% 36.04MB 0.72% github.com/Azure/azure-storage-blob-go/azblob.responderPolicy.Do
0 0% 96.60% 4722.57MB 94.49% github.com/rclone/rclone/backend/azureblob.(*Object).Update
0 0% 96.60% 4610.82MB 92.26% github.com/rclone/rclone/backend/azureblob.(*Object).Update.func1
0 0% 96.60% 64.51MB 1.29% github.com/rclone/rclone/backend/azureblob.(*Object).decodeMetaDataFromPropertiesResponse
0 0% 96.60% 112.02MB 2.24% github.com/rclone/rclone/backend/azureblob.(*Object).readMetaData
0 0% 96.60% 47.50MB 0.95% github.com/rclone/rclone/backend/azureblob.(*Object).readMetaData.func1
0 0% 96.60% 25.54MB 0.51% github.com/rclone/rclone/backend/azureblob.httpClientFactory.func1.1
0 0% 96.60% 35.86MB 0.72% github.com/rclone/rclone/cmd/mount.(*Dir).Lookup
0 0% 96.60% 36.83MB 0.74% github.com/rclone/rclone/cmd/mount.mount.func2
0 0% 96.60% 4669.32MB 93.43% github.com/rclone/rclone/fs.pacerInvoker
0 0% 96.60% 4738.57MB 94.81% github.com/rclone/rclone/fs/operations.Copy
0 0% 96.60% 58.78MB 1.18% github.com/rclone/rclone/lib/pacer.(*Pacer).Call
0 0% 96.60% 4609.55MB 92.23% github.com/rclone/rclone/lib/pacer.(*Pacer).CallNoRetry (inline)
0 0% 96.60% 4669.32MB 93.43% github.com/rclone/rclone/lib/pacer.(*Pacer).call
0 0% 96.60% 33.01MB 0.66% github.com/rclone/rclone/vfs.(*Dir).Create
0 0% 96.60% 35.86MB 0.72% github.com/rclone/rclone/vfs.(*Dir).Stat
0 0% 96.60% 37.86MB 0.76% github.com/rclone/rclone/vfs.(*Dir)._readDir
0 0% 96.60% 35.86MB 0.72% github.com/rclone/rclone/vfs.(*Dir).stat
0 0% 96.60% 95.75MB 1.92% github.com/rclone/rclone/vfs.(*File).Open
0 0% 96.60% 26.74MB 0.53% github.com/rclone/rclone/vfs.(*File).openRW
0 0% 96.60% 46.01MB 0.92% github.com/rclone/rclone/vfs/vfscache.(*Cache).Exists
0 0% 96.60% 46.01MB 0.92% github.com/rclone/rclone/vfs/vfscache.(*Cache)._get
0 0% 96.60% 46.01MB 0.92% github.com/rclone/rclone/vfs/vfscache.(*Cache).get
0 0% 96.60% 4720.57MB 94.45% github.com/rclone/rclone/vfs/vfscache.(*Item).Close.func2
0 0% 96.60% 4720.57MB 94.45% github.com/rclone/rclone/vfs/vfscache.(*Item)._store
0 0% 96.60% 4720.57MB 94.45% github.com/rclone/rclone/vfs/vfscache.(*Item).store
0 0% 96.60% 4720.57MB 94.45% github.com/rclone/rclone/vfs/vfscache/writeback.(*WriteBack).upload
0 0% 96.60% 4628.87MB 92.62% sync.(*Pool).Get
for comparison:
rclone version
rclone v1.52.3
uptime 3 days
htop:
1860391 root 20 0 12.7G 12.0G 9228 S 25.4 38.1 29h42:23 rclone mount --log-file=/var/log/dcmsys/rclone.log --log-level ERROR --config=/etc/rclone.conf --allow-other --default-permissions --max-read-ahead=4M --vfs-cache-mode=writes --vfs-cache-max-size=1G --azureblob-memory-pool-f
Fetching profile over HTTP from http://localhost:5572/debug/pprof/heap
Saved profile in /root/pprof/pprof.rclone.alloc_objects.alloc_space.inuse_objects.inuse_space.038.pb.gz
File: rclone
Type: inuse_space
Time: Jan 13, 2021 at 1:21am (EST)
Showing nodes accounting for 5.62GB, 98.02% of 5.73GB total
Dropped 98 nodes (cum <= 0.03GB)
flat flat% sum% cum cum%
0.95GB 16.58% 16.58% 0.95GB 16.58% github.com/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.BlobGetPropertiesResponse.NewMetadata
0.90GB 15.62% 32.20% 0.90GB 15.62% strings.(*Builder).grow (inline)
0.68GB 11.91% 44.12% 0.68GB 11.91% github.com/rclone/rclone/vfs.newFile
0.37GB 6.40% 50.52% 0.65GB 11.29% context.WithDeadline
0.34GB 5.95% 56.46% 0.34GB 5.95% net/textproto.(*Reader).ReadMIMEHeader
0.33GB 5.81% 62.28% 0.33GB 5.81% context.(*cancelCtx).Done
0.31GB 5.45% 67.73% 0.31GB 5.45% github.com/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.uploadStream.func1
0.31GB 5.42% 73.15% 2.60GB 45.39% github.com/rclone/rclone/backend/azureblob.(*Fs).Put
0.31GB 5.36% 78.51% 0.31GB 5.36% os.newFile
0.25GB 4.39% 82.90% 0.26GB 4.51% time.AfterFunc
0.22GB 3.86% 86.75% 0.22GB 3.86% github.com/rclone/rclone/vfs.(*Dir).addObject
0.19GB 3.40% 90.15% 1.29GB 22.52% github.com/rclone/rclone/vfs.newRWFileHandle
0.19GB 3.38% 93.54% 0.25GB 4.42% github.com/rclone/rclone/vendor/bazil.org/fuse.(*Conn).ReadRequest
0.09GB 1.54% 95.08% 0.09GB 1.54% encoding/base64.(*Encoding).EncodeToString
0.06GB 1.04% 96.12% 0.06GB 1.04% github.com/rclone/rclone/vendor/bazil.org/fuse.allocMessage
0.06GB 1.02% 97.14% 0.06GB 1.02% github.com/rclone/rclone/vfs.(*File).addWriter
0.03GB 0.56% 97.70% 0.03GB 0.56% time.startTimer
0.02GB 0.32% 98.02% 1.99GB 34.70% github.com/rclone/rclone/cmd/mount.(*Dir).Create
0 0% 98.02% 0.65GB 11.29% context.WithTimeout
0 0% 98.02% 0.04GB 0.63% encoding/xml.(*Decoder).Decode (inline)
0 0% 98.02% 0.04GB 0.63% encoding/xml.(*Decoder).DecodeElement
0 0% 98.02% 0.04GB 0.63% encoding/xml.(*Decoder).unmarshal
0 0% 98.02% 0.04GB 0.63% encoding/xml.(*Decoder).unmarshalPath
0 0% 98.02% 0.04GB 0.63% encoding/xml.Unmarshal
0 0% 98.02% 0.03GB 0.57% github.com/rclone/rclone/backend/azureblob.(*Fs).List
0 0% 98.02% 0.05GB 0.95% github.com/rclone/rclone/backend/azureblob.(*Fs).list
0 0% 98.02% 0.04GB 0.76% github.com/rclone/rclone/backend/azureblob.(*Fs).list.func1
0 0% 98.02% 0.03GB 0.6% github.com/rclone/rclone/backend/azureblob.(*Fs).listDir
0 0% 98.02% 2.32GB 40.53% github.com/rclone/rclone/backend/azureblob.(*Object).Update
0 0% 98.02% 0.45GB 7.82% github.com/rclone/rclone/backend/azureblob.(*Object).Update.func1
0 0% 98.02% 1.04GB 18.08% github.com/rclone/rclone/backend/azureblob.(*Object).decodeMetaDataFromPropertiesResponse
0 0% 98.02% 1.88GB 32.83% github.com/rclone/rclone/backend/azureblob.(*Object).readMetaData
0 0% 98.02% 0.85GB 14.90% github.com/rclone/rclone/backend/azureblob.(*Object).readMetaData.func1
0 0% 98.02% 0.33GB 5.83% github.com/rclone/rclone/backend/azureblob.httpClientFactory.func1.1
0 0% 98.02% 0.04GB 0.64% github.com/rclone/rclone/cmd/mount.(*Dir).Lookup
0 0% 98.02% 2.70GB 47.05% github.com/rclone/rclone/cmd/mount.(*FileHandle).Flush
0 0% 98.02% 0.25GB 4.42% github.com/rclone/rclone/cmd/mount.mount.func1
0 0% 98.02% 1.34GB 23.37% github.com/rclone/rclone/fs.pacerInvoker
0 0% 98.02% 0.33GB 5.83% github.com/rclone/rclone/fs/fshttp.(*Transport).RoundTrip
0 0% 98.02% 2.60GB 45.38% github.com/rclone/rclone/fs/operations.Copy
0 0% 98.02% 0.90GB 15.67% github.com/rclone/rclone/lib/pacer.(*Pacer).Call
0 0% 98.02% 0.44GB 7.65% github.com/rclone/rclone/lib/pacer.(*Pacer).CallNoRetry (inline)
0 0% 98.02% 1.34GB 23.37% github.com/rclone/rclone/lib/pacer.(*Pacer).call
0 0% 98.02% 0.06GB 1.04% github.com/rclone/rclone/vendor/bazil.org/fuse.getMessage (inline)
0 0% 98.02% 0.25GB 4.42% github.com/rclone/rclone/vendor/bazil.org/fuse/fs.(*Server).Serve
0 0% 98.02% 4.75GB 82.86% github.com/rclone/rclone/vendor/bazil.org/fuse/fs.(*Server).Serve.func1
0 0% 98.02% 4.75GB 82.86% github.com/rclone/rclone/vendor/bazil.org/fuse/fs.(*Server).handleRequest
0 0% 98.02% 4.75GB 82.86% github.com/rclone/rclone/vendor/bazil.org/fuse/fs.(*Server).serve
0 0% 98.02% 1.02GB 17.77% github.com/rclone/rclone/vendor/github.com/Azure/azure-pipeline-go/pipeline.(*pipeline).Do
0 0% 98.02% 1.02GB 17.77% github.com/rclone/rclone/vendor/github.com/Azure/azure-pipeline-go/pipeline.PolicyFunc.Do
0 0% 98.02% 0.37GB 6.48% github.com/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.(*SharedKeyCredential).New.func1
0 0% 98.02% 0.11GB 1.99% github.com/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.(*uploadStreamToBlockBlobOptions).end
0 0% 98.02% 0.85GB 14.90% github.com/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.BlobURL.GetProperties
0 0% 98.02% 0.11GB 1.98% github.com/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.BlockBlobURL.Upload
0 0% 98.02% 0.04GB 0.76% github.com/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.ContainerURL.ListBlobsHierarchySegment
0 0% 98.02% 0.33GB 5.83% github.com/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.NewRequestLogPolicyFactory.func1.1
0 0% 98.02% 1.02GB 17.77% github.com/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.NewRetryPolicyFactory.func1.1
0 0% 98.02% 1.02GB 17.77% github.com/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.NewTelemetryPolicyFactory.func1.1
0 0% 98.02% 1.02GB 17.77% github.com/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.NewUniqueRequestIDPolicyFactory.func1.1
0 0% 98.02% 0.45GB 7.82% github.com/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.UploadStreamToBlockBlob
0 0% 98.02% 0.85GB 14.90% github.com/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.blobClient.GetProperties
0 0% 98.02% 0.11GB 1.98% github.com/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.blockBlobClient.Upload
0 0% 98.02% 0.04GB 0.76% github.com/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.containerClient.ListBlobHierarchySegment
0 0% 98.02% 0.04GB 0.63% github.com/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.containerClient.listBlobHierarchySegmentResponder
0 0% 98.02% 0.37GB 6.46% github.com/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.responderPolicy.Do
0 0% 98.02% 0.45GB 7.82% github.com/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.uploadStream
0 0% 98.02% 0.68GB 11.85% github.com/rclone/rclone/vfs.(*Dir).Create
0 0% 98.02% 0.04GB 0.64% github.com/rclone/rclone/vfs.(*Dir).Stat
0 0% 98.02% 0.04GB 0.75% github.com/rclone/rclone/vfs.(*Dir)._readDir
0 0% 98.02% 0.04GB 0.64% github.com/rclone/rclone/vfs.(*Dir).stat
0 0% 98.02% 1.29GB 22.52% github.com/rclone/rclone/vfs.(*File).Open
0 0% 98.02% 0.37GB 6.52% github.com/rclone/rclone/vfs.(*File).Path
0 0% 98.02% 1.29GB 22.52% github.com/rclone/rclone/vfs.(*File).openRW
0 0% 98.02% 0.52GB 9.04% github.com/rclone/rclone/vfs.(*File).osPath
0 0% 98.02% 2.70GB 47.05% github.com/rclone/rclone/vfs.(*RWFileHandle).Flush
0 0% 98.02% 2.70GB 47.05% github.com/rclone/rclone/vfs.(*RWFileHandle).flushWrites
0 0% 98.02% 1.04GB 18.17% github.com/rclone/rclone/vfs.(*RWFileHandle).openPending
0 0% 98.02% 2.32GB 40.53% github.com/rclone/rclone/vfs/vfscache.(*Cache).Store
0 0% 98.02% 0.52GB 9.04% github.com/rclone/rclone/vfs/vfscache.(*Cache).ToOSPath (inline)
0 0% 98.02% 2.32GB 40.53% github.com/rclone/rclone/vfs/vfscache.copyObj
0 0% 98.02% 0.33GB 5.83% net/http.(*Client).Do (inline)
0 0% 98.02% 0.33GB 5.83% net/http.(*Client).do
0 0% 98.02% 0.33GB 5.83% net/http.(*Client).send
0 0% 98.02% 0.33GB 5.83% net/http.(*Transport).RoundTrip (inline)
0 0% 98.02% 0.33GB 5.83% net/http.(*Transport).roundTrip
0 0% 98.02% 0.34GB 5.95% net/http.(*persistConn).readLoop
0 0% 98.02% 0.34GB 5.95% net/http.(*persistConn).readResponse
0 0% 98.02% 0.34GB 5.95% net/http.ReadResponse
0 0% 98.02% 0.33GB 5.83% net/http.send
0 0% 98.02% 0.31GB 5.36% os.OpenFile
0 0% 98.02% 0.31GB 5.36% os.openFileNolog
0 0% 98.02% 0.38GB 6.55% path.Join
0 0% 98.02% 0.52GB 9.04% path/filepath.Join (inline)
0 0% 98.02% 0.52GB 9.04% path/filepath.join
0 0% 98.02% 0.90GB 15.62% strings.(*Builder).Grow (inline)
0 0% 98.02% 0.89GB 15.59% strings.Join
0 0% 98.02% 0.06GB 1.04% sync.(*Pool).Get
Can you try this one?
v1.54.0-beta.5061.705d00b8b.fix-azureblob-memory on branch fix-azureblob-memory (uploaded in 15-30 mins)
This contains one of the potential fixes to the azureblob SDK. That fix isn't merged yet but it looks good to me.
[root@AZ1VNA01 ~]# rclone version
rclone v1.54.0-beta.5061.705d00b8b.fix-azureblob-memory
- os/arch: linux/amd64
- go version: go1.15.6
[root@AZ1VNA01 ~]# go tool pprof -text http://localhost:5572/debug/pprof/heap
Fetching profile over HTTP from http://localhost:5572/debug/pprof/heap
Saved profile in /root/pprof/pprof.rclone.alloc_objects.alloc_space.inuse_objects.inuse_space.042.pb.gz
File: rclone
Type: inuse_space
Time: Jan 15, 2021 at 9:15am (EST)
Showing nodes accounting for 161.81MB, 96.71% of 167.32MB total
Dropped 42 nodes (cum <= 0.84MB)
flat flat% sum% cum cum%
28.47MB 17.01% 17.01% 28.47MB 17.01% bazil.org/fuse.allocMessage
23MB 13.75% 30.76% 23MB 13.75% github.com/rclone/rclone/vfs.newDir
20.51MB 12.26% 43.02% 20.51MB 12.26% github.com/Azure/azure-storage-blob-go/azblob.BlobGetPropertiesResponse.NewMetadata
9MB 5.38% 48.40% 13MB 7.77% github.com/rclone/rclone/vfs/vfscache.newItem
9MB 5.38% 53.78% 9MB 5.38% github.com/rclone/rclone/vfs.newFile
7.50MB 4.48% 58.26% 7.50MB 4.48% github.com/rclone/rclone/fs.NewDir (inline)
6.50MB 3.89% 62.15% 9.50MB 5.68% context.WithDeadline
6.22MB 3.72% 65.86% 6.22MB 3.72% github.com/rclone/rclone/vfs.(*Dir).addObject
6MB 3.59% 69.45% 41.51MB 24.81% github.com/rclone/rclone/backend/azureblob.(*Fs).Put
5.50MB 3.29% 72.74% 5.50MB 3.29% encoding/xml.copyValue
5MB 2.99% 75.73% 5MB 2.99% strings.(*Builder).grow (inline)
4.57MB 2.73% 78.46% 28.08MB 16.78% github.com/rclone/rclone/vfs.(*Dir)._readDirFromEntries
4.50MB 2.69% 81.15% 4.50MB 2.69% context.(*cancelCtx).Done
4MB 2.39% 83.54% 4MB 2.39% sync.NewCond (inline)
4MB 2.39% 85.93% 4MB 2.39% net/textproto.(*Reader).ReadMIMEHeader
3.50MB 2.09% 88.02% 31.97MB 19.11% bazil.org/fuse.(*Conn).ReadRequest
3MB 1.79% 89.82% 3MB 1.79% time.AfterFunc
3MB 1.79% 91.61% 9.72MB 5.81% github.com/rclone/rclone/vfs.newRWFileHandle
2.78MB 1.66% 93.27% 2.78MB 1.66% bazil.org/fuse/fs.(*Server).saveNode
2.50MB 1.49% 94.77% 39.48MB 23.59% github.com/rclone/rclone/cmd/mount.(*Dir).Create
1.50MB 0.9% 95.66% 1.50MB 0.9% time.NewTimer
1MB 0.6% 96.26% 1MB 0.6% context.WithCancel
0.75MB 0.45% 96.71% 13.75MB 8.22% github.com/rclone/rclone/vfs/vfscache.(*Cache)._get
0 0% 96.71% 28.47MB 17.01% bazil.org/fuse.getMessage (inline)
0 0% 96.71% 31.97MB 19.11% bazil.org/fuse/fs.(*Server).Serve
0 0% 96.71% 78.84MB 47.12% bazil.org/fuse/fs.(*Server).Serve.func1
0 0% 96.71% 78.84MB 47.12% bazil.org/fuse/fs.(*Server).handleRequest
0 0% 96.71% 2.78MB 1.66% bazil.org/fuse/fs.(*Server).saveLookup
0 0% 96.71% 78.84MB 47.12% bazil.org/fuse/fs.(*Server).serve
0 0% 96.71% 9.50MB 5.68% context.WithTimeout
0 0% 96.71% 1MB 0.6% crypto/tls.(*Conn).Handshake
0 0% 96.71% 1MB 0.6% crypto/tls.(*Conn).clientHandshake
0 0% 96.71% 1MB 0.6% crypto/tls.(*clientHandshakeState).doFullHandshake
0 0% 96.71% 1MB 0.6% crypto/tls.(*clientHandshakeState).handshake
0 0% 96.71% 5.50MB 3.29% encoding/xml.(*Decoder).Decode (inline)
0 0% 96.71% 5.50MB 3.29% encoding/xml.(*Decoder).DecodeElement
0 0% 96.71% 5.50MB 3.29% encoding/xml.(*Decoder).unmarshal
0 0% 96.71% 5.50MB 3.29% encoding/xml.(*Decoder).unmarshalPath
0 0% 96.71% 5.50MB 3.29% encoding/xml.Unmarshal
0 0% 96.71% 21MB 12.55% github.com/Azure/azure-pipeline-go/pipeline.(*pipeline).Do
0 0% 96.71% 21MB 12.55% github.com/Azure/azure-pipeline-go/pipeline.PolicyFunc.Do
0 0% 96.71% 11.50MB 6.87% github.com/Azure/azure-storage-blob-go/azblob.(*SharedKeyCredential).New.func1
0 0% 96.71% 1MB 0.6% github.com/Azure/azure-storage-blob-go/azblob.(*copier).close
0 0% 96.71% 1MB 0.6% github.com/Azure/azure-storage-blob-go/azblob.(*copier).write
0 0% 96.71% 1MB 0.6% github.com/Azure/azure-storage-blob-go/azblob.(*copier).writer
0 0% 96.71% 13MB 7.77% github.com/Azure/azure-storage-blob-go/azblob.BlobURL.GetProperties
0 0% 96.71% 1MB 0.6% github.com/Azure/azure-storage-blob-go/azblob.BlockBlobURL.CommitBlockList
0 0% 96.71% 1MB 0.6% github.com/Azure/azure-storage-blob-go/azblob.BlockBlobURL.StageBlock
0 0% 96.71% 6MB 3.59% github.com/Azure/azure-storage-blob-go/azblob.ContainerURL.ListBlobsHierarchySegment
0 0% 96.71% 6MB 3.59% github.com/Azure/azure-storage-blob-go/azblob.NewRequestLogPolicyFactory.func1.1
0 0% 96.71% 21MB 12.55% github.com/Azure/azure-storage-blob-go/azblob.NewRetryPolicyFactory.func1.1
0 0% 96.71% 21MB 12.55% github.com/Azure/azure-storage-blob-go/azblob.NewTelemetryPolicyFactory.func1.1
0 0% 96.71% 21MB 12.55% github.com/Azure/azure-storage-blob-go/azblob.NewUniqueRequestIDPolicyFactory.func1.1
0 0% 96.71% 2MB 1.20% github.com/Azure/azure-storage-blob-go/azblob.UploadStreamToBlockBlob
0 0% 96.71% 13MB 7.77% github.com/Azure/azure-storage-blob-go/azblob.blobClient.GetProperties
0 0% 96.71% 1MB 0.6% github.com/Azure/azure-storage-blob-go/azblob.blockBlobClient.CommitBlockList
0 0% 96.71% 1MB 0.6% github.com/Azure/azure-storage-blob-go/azblob.blockBlobClient.StageBlock
0 0% 96.71% 6MB 3.59% github.com/Azure/azure-storage-blob-go/azblob.containerClient.ListBlobHierarchySegment
0 0% 96.71% 5.50MB 3.29% github.com/Azure/azure-storage-blob-go/azblob.containerClient.listBlobHierarchySegmentResponder
0 0% 96.71% 2MB 1.20% github.com/Azure/azure-storage-blob-go/azblob.copyFromReader
0 0% 96.71% 11.50MB 6.87% github.com/Azure/azure-storage-blob-go/azblob.responderPolicy.Do
0 0% 96.71% 14MB 8.37% github.com/rclone/rclone/backend/azureblob.(*Fs).List
0 0% 96.71% 8MB 4.78% github.com/rclone/rclone/backend/azureblob.(*Fs).itemToDirEntry
0 0% 96.71% 14MB 8.37% github.com/rclone/rclone/backend/azureblob.(*Fs).list
0 0% 96.71% 6MB 3.59% github.com/rclone/rclone/backend/azureblob.(*Fs).list.func1
0 0% 96.71% 14MB 8.37% github.com/rclone/rclone/backend/azureblob.(*Fs).listDir
0 0% 96.71% 8MB 4.78% github.com/rclone/rclone/backend/azureblob.(*Fs).listDir.func1
0 0% 96.71% 36.01MB 21.52% github.com/rclone/rclone/backend/azureblob.(*Object).Update
0 0% 96.71% 2MB 1.20% github.com/rclone/rclone/backend/azureblob.(*Object).Update.func1
0 0% 96.71% 21.01MB 12.55% github.com/rclone/rclone/backend/azureblob.(*Object).decodeMetaDataFromPropertiesResponse
0 0% 96.71% 34.01MB 20.32% github.com/rclone/rclone/backend/azureblob.(*Object).readMetaData
0 0% 96.71% 13MB 7.77% github.com/rclone/rclone/backend/azureblob.(*Object).readMetaData.func1
0 0% 96.71% 6MB 3.59% github.com/rclone/rclone/backend/azureblob.httpClientFactory.func1.1
0 0% 96.71% 35.58MB 21.26% github.com/rclone/rclone/cmd/mount.(*Dir).Lookup
0 0% 96.71% 1MB 0.6% github.com/rclone/rclone/cmd/mount.(*Dir).ReadDirAll
0 0% 96.71% 31.97MB 19.11% github.com/rclone/rclone/cmd/mount.mount.func2
0 0% 96.71% 21MB 12.55% github.com/rclone/rclone/fs.pacerInvoker
0 0% 96.71% 6MB 3.59% github.com/rclone/rclone/fs/fshttp.(*Transport).RoundTrip
0 0% 96.71% 8.50MB 5.08% github.com/rclone/rclone/fs/list.DirSorted
0 0% 96.71% 41.51MB 24.81% github.com/rclone/rclone/fs/operations.Copy
0 0% 96.71% 19MB 11.36% github.com/rclone/rclone/lib/pacer.(*Pacer).Call
0 0% 96.71% 2MB 1.20% github.com/rclone/rclone/lib/pacer.(*Pacer).CallNoRetry (inline)
0 0% 96.71% 21MB 12.55% github.com/rclone/rclone/lib/pacer.(*Pacer).call
0 0% 96.71% 8.50MB 5.08% github.com/rclone/rclone/vfs.(*Dir).Create
0 0% 96.71% 1MB 0.6% github.com/rclone/rclone/vfs.(*Dir).ReadDirAll
0 0% 96.71% 35.58MB 21.26% github.com/rclone/rclone/vfs.(*Dir).Stat
0 0% 96.71% 36.58MB 21.86% github.com/rclone/rclone/vfs.(*Dir)._readDir
0 0% 96.71% 35.58MB 21.26% github.com/rclone/rclone/vfs.(*Dir).stat
0 0% 96.71% 28.48MB 17.02% github.com/rclone/rclone/vfs.(*File).Open
0 0% 96.71% 5MB 2.99% github.com/rclone/rclone/vfs.(*File).Path
0 0% 96.71% 9.72MB 5.81% github.com/rclone/rclone/vfs.(*File).openRW
0 0% 96.71% 6.22MB 3.72% github.com/rclone/rclone/vfs.(*RWFileHandle).Truncate
0 0% 96.71% 6.22MB 3.72% github.com/rclone/rclone/vfs.(*RWFileHandle).openPending
0 0% 96.71% 13.75MB 8.22% github.com/rclone/rclone/vfs/vfscache.(*Cache).Exists
0 0% 96.71% 13.75MB 8.22% github.com/rclone/rclone/vfs/vfscache.(*Cache).get
0 0% 96.71% 37.51MB 22.42% github.com/rclone/rclone/vfs/vfscache.(*Item).Close.func2
0 0% 96.71% 37.51MB 22.42% github.com/rclone/rclone/vfs/vfscache.(*Item)._store
0 0% 96.71% 37.51MB 22.42% github.com/rclone/rclone/vfs/vfscache.(*Item).store
0 0% 96.71% 37.51MB 22.42% github.com/rclone/rclone/vfs/vfscache/writeback.(*WriteBack).upload
0 0% 96.71% 6MB 3.59% net/http.(*Client).Do (inline)
0 0% 96.71% 6MB 3.59% net/http.(*Client).do
0 0% 96.71% 6MB 3.59% net/http.(*Client).send
0 0% 96.71% 6MB 3.59% net/http.(*Transport).RoundTrip (inline)
0 0% 96.71% 6MB 3.59% net/http.(*Transport).roundTrip
0 0% 96.71% 1MB 0.6% net/http.(*persistConn).addTLS.func2
0 0% 96.71% 4MB 2.39% net/http.(*persistConn).readLoop
0 0% 96.71% 4MB 2.39% net/http.(*persistConn).readResponse
0 0% 96.71% 1.50MB 0.9% net/http.(*persistConn).roundTrip
0 0% 96.71% 4MB 2.39% net/http.ReadResponse
0 0% 96.71% 6MB 3.59% net/http.send
0 0% 96.71% 5MB 2.99% path.Join
0 0% 96.71% 1.50MB 0.9% runtime.doInit
0 0% 96.71% 1.50MB 0.9% runtime.main
0 0% 96.71% 5MB 2.99% strings.(*Builder).Grow (inline)
0 0% 96.71% 5MB 2.99% strings.Join
0 0% 96.71% 28.47MB 17.01% sync.(*Pool).Get
Uptime 2h
HTOP:
3266535 root 20 0 1081M 418M 31500 S 6.6 1.3 16:32.95 rclone mount --log-file=/var/log/dcmsys/rclone.log --log-level ERROR --config=/etc/rclone.conf --allow-other --default-permissions --max-read-ahead=4M --vfs-cache-mode=writes --vfs-cache-max-size=1G --azureblob-memory-pool-f
That looks much better - 20MB in use by the azblob sdk.
I need this to be merged upstream before I can include it in rclone, but hopefully that will be in time for 1.54
uptime 3 days
rclone version
rclone v1.54.0-beta.5061.705d00b8b.fix-azureblob-memory
- os/arch: linux/amd64
- go version: go1.15.6
HTOP:
3266535 root 20 0 4105M 3255M 10164 S 0.0 10.1 11h02:02 rclone mount --log-file=/var/log/dcmsys/rclone.log --log-level ERROR --config=/etc/rclone.conf --allow-other --default-permissions --max-read-ahead=4M --vfs-cache-mode=writes --vfs-ca
Fetching profile over HTTP from http://localhost:5572/debug/pprof/heap
Saved profile in /root/pprof/pprof.rclone.alloc_objects.alloc_space.inuse_objects.inuse_space.044.pb.gz
File: rclone
Type: inuse_space
Time: Jan 18, 2021 at 10:20am (EST)
Showing nodes accounting for 2586.17MB, 98.67% of 2620.96MB total
Dropped 52 nodes (cum <= 13.10MB)
flat flat% sum% cum cum%
451.11MB 17.21% 17.21% 451.61MB 17.23% github.com/Azure/azure-storage-blob-go/azblob.BlobGetPropertiesResponse.NewMetadata
325.57MB 12.42% 29.63% 406.57MB 15.51% github.com/rclone/rclone/vfs/vfscache.newItem
252.55MB 9.64% 39.27% 252.55MB 9.64% github.com/rclone/rclone/vfs.newFile
187.52MB 7.15% 46.42% 187.52MB 7.15% context.(*cancelCtx).Done
186.52MB 7.12% 53.54% 333.13MB 12.71% context.WithDeadline
184.52MB 7.04% 60.58% 184.52MB 7.04% strings.(*Builder).grow (inline)
159.31MB 6.08% 66.66% 159.31MB 6.08% github.com/rclone/rclone/vfs.(*Dir).addObject
155.01MB 5.91% 72.57% 155.01MB 5.91% net/textproto.(*Reader).ReadMIMEHeader
144.52MB 5.51% 78.09% 1063.71MB 40.58% github.com/rclone/rclone/backend/azureblob.(*Fs).Put
124.01MB 4.73% 82.82% 131.11MB 5.00% time.AfterFunc
88.51MB 3.38% 86.19% 88.51MB 3.38% bazil.org/fuse.(*Conn).ReadRequest
81MB 3.09% 89.29% 81MB 3.09% sync.NewCond (inline)
78MB 2.98% 92.26% 255.81MB 9.76% github.com/rclone/rclone/vfs.newRWFileHandle
41MB 1.56% 93.83% 41MB 1.56% encoding/base64.(*Encoding).EncodeToString
36.51MB 1.39% 95.22% 36.51MB 1.39% github.com/rclone/rclone/vfs.newDir
25.50MB 0.97% 96.19% 1112.70MB 42.45% github.com/rclone/rclone/cmd/mount.(*Dir).Create
22.50MB 0.86% 97.05% 35.50MB 1.35% context.WithCancel
18.50MB 0.71% 97.76% 18.50MB 0.71% github.com/rclone/rclone/vfs.(*File).addWriter
15.09MB 0.58% 98.33% 15.09MB 0.58% time.startTimer
8.19MB 0.31% 98.64% 43.69MB 1.67% github.com/rclone/rclone/vfs.(*Dir)._readDirFromEntries
0.75MB 0.029% 98.67% 407.32MB 15.54% github.com/rclone/rclone/vfs/vfscache.(*Cache)._get
0 0% 98.67% 88.51MB 3.38% bazil.org/fuse/fs.(*Server).Serve
0 0% 98.67% 1192.68MB 45.51% bazil.org/fuse/fs.(*Server).Serve.func1
0 0% 98.67% 1192.68MB 45.51% bazil.org/fuse/fs.(*Server).handleRequest
0 0% 98.67% 1192.68MB 45.51% bazil.org/fuse/fs.(*Server).serve
0 0% 98.67% 333.13MB 12.71% context.WithTimeout
0 0% 98.67% 28.50MB 1.09% context.propagateCancel
0 0% 98.67% 15MB 0.57% encoding/xml.(*Decoder).Decode (inline)
0 0% 98.67% 15MB 0.57% encoding/xml.(*Decoder).DecodeElement
0 0% 98.67% 15MB 0.57% encoding/xml.(*Decoder).unmarshal
0 0% 98.67% 15MB 0.57% encoding/xml.(*Decoder).unmarshalPath
0 0% 98.67% 15MB 0.57% encoding/xml.Unmarshal
0 0% 98.67% 511.58MB 19.52% github.com/Azure/azure-pipeline-go/pipeline.(*pipeline).Do
0 0% 98.67% 511.58MB 19.52% github.com/Azure/azure-pipeline-go/pipeline.PolicyFunc.Do
0 0% 98.67% 178.45MB 6.81% github.com/Azure/azure-storage-blob-go/azblob.(*SharedKeyCredential).New.func1
0 0% 98.67% 31.55MB 1.20% github.com/Azure/azure-storage-blob-go/azblob.(*copier).close
0 0% 98.67% 59.49MB 2.27% github.com/Azure/azure-storage-blob-go/azblob.(*copier).write
0 0% 98.67% 59.49MB 2.27% github.com/Azure/azure-storage-blob-go/azblob.(*copier).writer
0 0% 98.67% 397.53MB 15.17% github.com/Azure/azure-storage-blob-go/azblob.BlobURL.GetProperties
0 0% 98.67% 31.55MB 1.20% github.com/Azure/azure-storage-blob-go/azblob.BlockBlobURL.CommitBlockList
0 0% 98.67% 59.49MB 2.27% github.com/Azure/azure-storage-blob-go/azblob.BlockBlobURL.StageBlock
0 0% 98.67% 23MB 0.88% github.com/Azure/azure-storage-blob-go/azblob.ContainerURL.ListBlobsHierarchySegment
0 0% 98.67% 163.45MB 6.24% github.com/Azure/azure-storage-blob-go/azblob.NewRequestLogPolicyFactory.func1.1
0 0% 98.67% 511.58MB 19.52% github.com/Azure/azure-storage-blob-go/azblob.NewRetryPolicyFactory.func1.1
0 0% 98.67% 511.58MB 19.52% github.com/Azure/azure-storage-blob-go/azblob.NewTelemetryPolicyFactory.func1.1
0 0% 98.67% 511.58MB 19.52% github.com/Azure/azure-storage-blob-go/azblob.NewUniqueRequestIDPolicyFactory.func1.1
0 0% 98.67% 54.55MB 2.08% github.com/Azure/azure-storage-blob-go/azblob.UploadStreamToBlockBlob
0 0% 98.67% 397.53MB 15.17% github.com/Azure/azure-storage-blob-go/azblob.blobClient.GetProperties
0 0% 98.67% 31.55MB 1.20% github.com/Azure/azure-storage-blob-go/azblob.blockBlobClient.CommitBlockList
0 0% 98.67% 59.49MB 2.27% github.com/Azure/azure-storage-blob-go/azblob.blockBlobClient.StageBlock
0 0% 98.67% 23MB 0.88% github.com/Azure/azure-storage-blob-go/azblob.containerClient.ListBlobHierarchySegment
0 0% 98.67% 15MB 0.57% github.com/Azure/azure-storage-blob-go/azblob.containerClient.listBlobHierarchySegmentResponder
0 0% 98.67% 54.55MB 2.08% github.com/Azure/azure-storage-blob-go/azblob.copyFromReader
0 0% 98.67% 178.45MB 6.81% github.com/Azure/azure-storage-blob-go/azblob.responderPolicy.Do
0 0% 98.67% 30.50MB 1.16% github.com/rclone/rclone/backend/azureblob.(*Fs).List
0 0% 98.67% 35MB 1.34% github.com/rclone/rclone/backend/azureblob.(*Fs).list
0 0% 98.67% 23MB 0.88% github.com/rclone/rclone/backend/azureblob.(*Fs).list.func1
0 0% 98.67% 31.50MB 1.20% github.com/rclone/rclone/backend/azureblob.(*Fs).listDir
0 0% 98.67% 944.70MB 36.04% github.com/rclone/rclone/backend/azureblob.(*Object).Update
0 0% 98.67% 54.55MB 2.08% github.com/rclone/rclone/backend/azureblob.(*Object).Update.func1
0 0% 98.67% 492.61MB 18.80% github.com/rclone/rclone/backend/azureblob.(*Object).decodeMetaDataFromPropertiesResponse
0 0% 98.67% 890.14MB 33.96% github.com/rclone/rclone/backend/azureblob.(*Object).readMetaData
0 0% 98.67% 397.53MB 15.17% github.com/rclone/rclone/backend/azureblob.(*Object).readMetaData.func1
0 0% 98.67% 163.45MB 6.24% github.com/rclone/rclone/backend/azureblob.httpClientFactory.func1.1
0 0% 98.67% 56.13MB 2.14% github.com/rclone/rclone/cmd/mount.(*Dir).Lookup
0 0% 98.67% 88.51MB 3.38% github.com/rclone/rclone/cmd/mount.mount.func2
0 0% 98.67% 474.59MB 18.11% github.com/rclone/rclone/fs.pacerInvoker
0 0% 98.67% 163.45MB 6.24% github.com/rclone/rclone/fs/fshttp.(*Transport).RoundTrip
0 0% 98.67% 20.50MB 0.78% github.com/rclone/rclone/fs/list.DirSorted
0 0% 98.67% 1080.21MB 41.21% github.com/rclone/rclone/fs/operations.Copy
0 0% 98.67% 420.04MB 16.03% github.com/rclone/rclone/lib/pacer.(*Pacer).Call
0 0% 98.67% 54.55MB 2.08% github.com/rclone/rclone/lib/pacer.(*Pacer).CallNoRetry (inline)
0 0% 98.67% 474.59MB 18.11% github.com/rclone/rclone/lib/pacer.(*Pacer).call
0 0% 98.67% 250.55MB 9.56% github.com/rclone/rclone/vfs.(*Dir).Create
0 0% 98.67% 56.13MB 2.14% github.com/rclone/rclone/vfs.(*Dir).Stat
0 0% 98.67% 61.69MB 2.35% github.com/rclone/rclone/vfs.(*Dir)._readDir
0 0% 98.67% 56.13MB 2.14% github.com/rclone/rclone/vfs.(*Dir).stat
0 0% 98.67% 845.66MB 32.27% github.com/rclone/rclone/vfs.(*File).Open
0 0% 98.67% 182.52MB 6.96% github.com/rclone/rclone/vfs.(*File).Path
0 0% 98.67% 255.81MB 9.76% github.com/rclone/rclone/vfs.(*File).openRW
0 0% 98.67% 159.31MB 6.08% github.com/rclone/rclone/vfs.(*RWFileHandle).Truncate
0 0% 98.67% 159.31MB 6.08% github.com/rclone/rclone/vfs.(*RWFileHandle).openPending
0 0% 98.67% 407.32MB 15.54% github.com/rclone/rclone/vfs/vfscache.(*Cache).Exists
0 0% 98.67% 407.32MB 15.54% github.com/rclone/rclone/vfs/vfscache.(*Cache).get
0 0% 98.67% 945.20MB 36.06% github.com/rclone/rclone/vfs/vfscache.(*Item).Close.func2
0 0% 98.67% 949.70MB 36.23% github.com/rclone/rclone/vfs/vfscache.(*Item)._store
0 0% 98.67% 945.20MB 36.06% github.com/rclone/rclone/vfs/vfscache.(*Item).store
0 0% 98.67% 945.20MB 36.06% github.com/rclone/rclone/vfs/vfscache/writeback.(*WriteBack).upload
0 0% 98.67% 163.45MB 6.24% net/http.(*Client).Do (inline)
0 0% 98.67% 163.45MB 6.24% net/http.(*Client).do
0 0% 98.67% 163.45MB 6.24% net/http.(*Client).send
0 0% 98.67% 163.45MB 6.24% net/http.(*Transport).RoundTrip (inline)
0 0% 98.67% 163.45MB 6.24% net/http.(*Transport).roundTrip
0 0% 98.67% 155.01MB 5.91% net/http.(*persistConn).readLoop
0 0% 98.67% 155.01MB 5.91% net/http.(*persistConn).readResponse
0 0% 98.67% 155.01MB 5.91% net/http.ReadResponse
0 0% 98.67% 163.45MB 6.24% net/http.send
0 0% 98.67% 184.02MB 7.02% path.Join
0 0% 98.67% 184.52MB 7.04% strings.(*Builder).Grow (inline)
0 0% 98.67% 184.02MB 7.02% strings.Join
Is that acceptable?
Your profile looks like you have lots of objects stored in the VFS cache - that is where the memory is being used I think.
4Gb of ram is OK for me. I need to be sure that the memory consumption won't grow constantly without limits. If consuming won't exceed 8Gb for example, then it's ok.
It is looking OK now I think
Hi Nick,
Not really.
After 9 days of working I see that rclone uses 13Gb of RAM. Could you please recommend me what can I do in order to stop memory consumption going up? I need to have stable RAM consumption. 8Gb is OK, but not more.
uptime: 1 week 2 days
htop:
3266535 root 20 0 12.9G 12.3G 11472 S 362. 39.0 67h34:41 rclone mount --log-file=/var/log/dcmsys/rclone.log --log-level ERROR --config=/etc/rclone.conf --allow-other --default-permissions --max-read-ahead=4M --vfs-cache-mode=writes -
rclone mount --log-file=/var/log/dcmsys/rclone.log --log-level ERROR --config=/etc/rclone.conf --allow-other --default-permissions --max-read-ahead=4M --vfs-cache-mode=writes --vfs-cache-max-size=1G --azureblob-memory-pool-flush-time=1m0s --buffer-size=0 --azureblob-memory-pool-use-mmap --use-mmap --dir-cache-time=2h --timeout=1h --cache-dir=/var/dcmsys/cache --uid=1100 --gid=1100 --cache-chunk-no-memory --rc --size-only --allow-non-empty blob:studiesh /var/dcmsys/storages/GCP
go tool pprof -text http://localhost:5572/debug/pprof/heap
Fetching profile over HTTP from http://localhost:5572/debug/pprof/heap
Saved profile in /root/pprof/pprof.rclone.alloc_objects.alloc_space.inuse_objects.inuse_space.046.pb.gz
File: rclone
Type: inuse_space
Time: Jan 24, 2021 at 2:55pm (EST)
Showing nodes accounting for 10948.52MB, 98.53% of 11111.98MB total
Dropped 200 nodes (cum <= 55.56MB)
flat flat% sum% cum cum%
1983.98MB 17.85% 17.85% 1984.48MB 17.86% github.com/Azure/azure-storage-blob-go/azblob.BlobGetPropertiesResponse.NewMetadata
1379.29MB 12.41% 30.27% 1762.82MB 15.86% github.com/rclone/rclone/vfs/vfscache.newItem
1133.71MB 10.20% 40.47% 1133.71MB 10.20% github.com/rclone/rclone/vfs.newFile
821.60MB 7.39% 47.86% 821.60MB 7.39% strings.(*Builder).grow (inline)
717.56MB 6.46% 54.32% 1271.10MB 11.44% context.WithDeadline
690.06MB 6.21% 60.53% 690.06MB 6.21% context.(*cancelCtx).Done
678.52MB 6.11% 66.64% 678.52MB 6.11% net/textproto.(*Reader).ReadMIMEHeader
674.57MB 6.07% 72.71% 4686.06MB 42.17% github.com/rclone/rclone/backend/azureblob.(*Fs).Put
663.85MB 5.97% 78.68% 663.85MB 5.97% github.com/rclone/rclone/vfs.(*Dir).addObject
525.54MB 4.73% 83.41% 539.09MB 4.85% time.AfterFunc
416.53MB 3.75% 87.16% 427.84MB 3.85% bazil.org/fuse.(*Conn).ReadRequest
407.52MB 3.67% 90.83% 1156.83MB 10.41% github.com/rclone/rclone/vfs.newRWFileHandle
382.52MB 3.44% 94.27% 382.52MB 3.44% sync.NewCond (inline)
194.51MB 1.75% 96.02% 194.51MB 1.75% encoding/base64.(*Encoding).EncodeToString
93.50MB 0.84% 96.86% 4940.71MB 44.46% github.com/rclone/rclone/cmd/mount.(*Dir).Create
86MB 0.77% 97.64% 86MB 0.77% github.com/rclone/rclone/vfs.(*File).addWriter
61MB 0.55% 98.18% 61MB 0.55% time.startTimer
33MB 0.3% 98.48% 61.50MB 0.55% context.WithCancel
1.25MB 0.011% 98.49% 1764.07MB 15.88% github.com/rclone/rclone/vfs/vfscache.(*Cache)._get
1MB 0.009% 98.50% 702.46MB 6.32% github.com/Azure/azure-storage-blob-go/azblob.(*SharedKeyCredential).New.func1
1MB 0.009% 98.51% 119.52MB 1.08% github.com/Azure/azure-storage-blob-go/azblob.copyFromReader
1MB 0.009% 98.52% 5066.74MB 45.60% bazil.org/fuse/fs.(*Server).handleRequest
1MB 0.009% 98.53% 1769.09MB 15.92% github.com/Azure/azure-storage-blob-go/azblob.blobClient.GetProperties
0 0% 98.53% 427.84MB 3.85% bazil.org/fuse/fs.(*Server).Serve
0 0% 98.53% 5067.24MB 45.60% bazil.org/fuse/fs.(*Server).Serve.func1
0 0% 98.53% 5067.24MB 45.60% bazil.org/fuse/fs.(*Server).serve
0 0% 98.53% 1271.10MB 11.44% context.WithTimeout
0 0% 98.53% 57.01MB 0.51% context.propagateCancel
0 0% 98.53% 1978.06MB 17.80% github.com/Azure/azure-pipeline-go/pipeline.(*pipeline).Do
0 0% 98.53% 1976.56MB 17.79% github.com/Azure/azure-pipeline-go/pipeline.PolicyFunc.Do
0 0% 98.53% 70.51MB 0.63% github.com/Azure/azure-storage-blob-go/azblob.(*copier).close
0 0% 98.53% 98.45MB 0.89% github.com/Azure/azure-storage-blob-go/azblob.(*copier).write
0 0% 98.53% 98.45MB 0.89% github.com/Azure/azure-storage-blob-go/azblob.(*copier).writer
0 0% 98.53% 1769.09MB 15.92% github.com/Azure/azure-storage-blob-go/azblob.BlobURL.GetProperties
0 0% 98.53% 70.51MB 0.63% github.com/Azure/azure-storage-blob-go/azblob.BlockBlobURL.CommitBlockList
0 0% 98.53% 98.45MB 0.89% github.com/Azure/azure-storage-blob-go/azblob.BlockBlobURL.StageBlock
0 0% 98.53% 674.45MB 6.07% github.com/Azure/azure-storage-blob-go/azblob.NewRequestLogPolicyFactory.func1.1
0 0% 98.53% 1976.06MB 17.78% github.com/Azure/azure-storage-blob-go/azblob.NewRetryPolicyFactory.func1.1
0 0% 98.53% 1976.56MB 17.79% github.com/Azure/azure-storage-blob-go/azblob.NewTelemetryPolicyFactory.func1.1
0 0% 98.53% 1976.56MB 17.79% github.com/Azure/azure-storage-blob-go/azblob.NewUniqueRequestIDPolicyFactory.func1.1
0 0% 98.53% 119.52MB 1.08% github.com/Azure/azure-storage-blob-go/azblob.UploadStreamToBlockBlob
0 0% 98.53% 70.51MB 0.63% github.com/Azure/azure-storage-blob-go/azblob.blockBlobClient.CommitBlockList
0 0% 98.53% 98.45MB 0.89% github.com/Azure/azure-storage-blob-go/azblob.blockBlobClient.StageBlock
0 0% 98.53% 698.45MB 6.29% github.com/Azure/azure-storage-blob-go/azblob.responderPolicy.Do
0 0% 98.53% 60.51MB 0.54% github.com/rclone/rclone/backend/azureblob.(*Fs).list
0 0% 98.53% 4062.49MB 36.56% github.com/rclone/rclone/backend/azureblob.(*Object).Update
0 0% 98.53% 119.52MB 1.08% github.com/rclone/rclone/backend/azureblob.(*Object).Update.func1
0 0% 98.53% 2178.48MB 19.60% github.com/rclone/rclone/backend/azureblob.(*Object).decodeMetaDataFromPropertiesResponse
0 0% 98.53% 3930.63MB 35.37% github.com/rclone/rclone/backend/azureblob.(*Object).readMetaData
0 0% 98.53% 1769.09MB 15.92% github.com/rclone/rclone/backend/azureblob.(*Object).readMetaData.func1
0 0% 98.53% 674.45MB 6.07% github.com/rclone/rclone/backend/azureblob.httpClientFactory.func1.1
0 0% 98.53% 69.14MB 0.62% github.com/rclone/rclone/cmd/mount.(*Dir).Lookup
0 0% 98.53% 427.84MB 3.85% github.com/rclone/rclone/cmd/mount.mount.func2
0 0% 98.53% 1933.11MB 17.40% github.com/rclone/rclone/fs.pacerInvoker
0 0% 98.53% 672.95MB 6.06% github.com/rclone/rclone/fs/fshttp.(*Transport).RoundTrip
0 0% 98.53% 4717.06MB 42.45% github.com/rclone/rclone/fs/operations.Copy
0 0% 98.53% 1813.10MB 16.32% github.com/rclone/rclone/lib/pacer.(*Pacer).Call
0 0% 98.53% 118.52MB 1.07% github.com/rclone/rclone/lib/pacer.(*Pacer).CallNoRetry (inline)
0 0% 98.53% 1933.11MB 17.40% github.com/rclone/rclone/lib/pacer.(*Pacer).call
0 0% 98.53% 1131.21MB 10.18% github.com/rclone/rclone/vfs.(*Dir).Create
0 0% 98.53% 69.14MB 0.62% github.com/rclone/rclone/vfs.(*Dir).Stat
0 0% 98.53% 77.70MB 0.7% github.com/rclone/rclone/vfs.(*Dir)._readDir
0 0% 98.53% 69.14MB 0.62% github.com/rclone/rclone/vfs.(*Dir).stat
0 0% 98.53% 3732.50MB 33.59% github.com/rclone/rclone/vfs.(*File).Open
0 0% 98.53% 811.60MB 7.30% github.com/rclone/rclone/vfs.(*File).Path
0 0% 98.53% 1156.83MB 10.41% github.com/rclone/rclone/vfs.(*File).openRW
0 0% 98.53% 663.33MB 5.97% github.com/rclone/rclone/vfs.(*RWFileHandle).Truncate
0 0% 98.53% 663.33MB 5.97% github.com/rclone/rclone/vfs.(*RWFileHandle).openPending
0 0% 98.53% 1764.07MB 15.88% github.com/rclone/rclone/vfs/vfscache.(*Cache).Exists
0 0% 98.53% 1764.07MB 15.88% github.com/rclone/rclone/vfs/vfscache.(*Cache).get
0 0% 98.53% 4125.51MB 37.13% github.com/rclone/rclone/vfs/vfscache.(*Item).Close.func2
0 0% 98.53% 4132.51MB 37.19% github.com/rclone/rclone/vfs/vfscache.(*Item)._store
0 0% 98.53% 4125.51MB 37.13% github.com/rclone/rclone/vfs/vfscache.(*Item).store
0 0% 98.53% 4125.51MB 37.13% github.com/rclone/rclone/vfs/vfscache/writeback.(*WriteBack).upload
0 0% 98.53% 674.45MB 6.07% net/http.(*Client).Do (inline)
0 0% 98.53% 674.45MB 6.07% net/http.(*Client).do
0 0% 98.53% 672.95MB 6.06% net/http.(*Client).send
0 0% 98.53% 672.95MB 6.06% net/http.(*Transport).RoundTrip (inline)
0 0% 98.53% 672.95MB 6.06% net/http.(*Transport).roundTrip
0 0% 98.53% 679.02MB 6.11% net/http.(*persistConn).readLoop
0 0% 98.53% 679.02MB 6.11% net/http.(*persistConn).readResponse
0 0% 98.53% 679.02MB 6.11% net/http.ReadResponse
0 0% 98.53% 672.95MB 6.06% net/http.send
0 0% 98.53% 818.60MB 7.37% path.Join
0 0% 98.53% 821.60MB 7.39% strings.(*Builder).Grow (inline)
0 0% 98.53% 821.10MB 7.39% strings.Join
Best,
Raymond
What you can see in this memory profile is that the memory is used by the VFS cache.
So it looks like the azure blob SDK leak is fixed, but you have a lot of cached items in the VFS cache.
I think this is the same issue as
How many items do you have in the VFS cache?