Ask for settings recommendation for azureblob

Files are from 16Kb and up to 500Mb each. Average - 2-3Mb each.
Usage - both reading and writing. Usually - reading. Writing in most cases - only one time.
I'll try to produce a memory profile tomorrow. I cannot unmount the storage now.

1 Like

system is running for 2 hours.
According to the htop, rclone took about 8Gb of RAM.
1533383 root 20 0 8733M 7970M 31492 S 15.3 24.8 33:15.72 rclone mount --log-f...

rclone mount --log-file=/var/log/dcmsys/rclone.log --log-level ERROR --config=/etc/rclone.conf --allow-other --default-permissions --max-read-ahead=4M --vfs-cache-mode=writes --vfs-cache-max-size=1G --azureblob-memory-pool-flush-time=1m0s --buffer-size=0 --azureblob-memory-pool-use-mmap --use-mmap --dir-cache-time=2h --timeout=1h --cache-dir=/var/dcmsys/cache --uid=1100 --gid=1100 --cache-chunk-no-memory --rc --size-only --allow-non-empty blob:studiesh /var/dcmsys/storages/GCP

I have file pprof.rclone.alloc_objects.alloc_space.inuse_objects.inuse_space.026.pb.gz but not sure how to share it here.

Fetching profile over HTTP from http://localhost:5572/debug/pprof/heap
Saved profile in /root/pprof/pprof.rclone.alloc_objects.alloc_space.inuse_objects.inuse_space.026.pb.gz
File: rclone
Type: inuse_space
Time: Jan 8, 2021 at 11:10am (EST)
Showing nodes accounting for 3758.28MB, 96.70% of 3886.48MB total
Dropped 131 nodes (cum <= 19.43MB)
      flat  flat%   sum%        cum   cum%
 3605.21MB 92.76% 92.76%  3605.21MB 92.76%  github.com/Azure/azure-storage-blob-go/azblob.copyFromReader.func1
   40.51MB  1.04% 93.81%    40.51MB  1.04%  github.com/Azure/azure-storage-blob-go/azblob.BlobGetPropertiesResponse.NewMetadata
      25MB  0.64% 94.45%       25MB  0.64%  github.com/rclone/rclone/vfs.newFile
   24.01MB  0.62% 95.07%    29.01MB  0.75%  github.com/rclone/rclone/vfs/vfscache.newItem
      20MB  0.51% 95.58%       20MB  0.51%  strings.(*Builder).grow (inline)
      14MB  0.36% 95.94%    23.71MB  0.61%  context.WithDeadline
   12.50MB  0.32% 96.26%  3327.12MB 85.61%  github.com/rclone/rclone/backend/azureblob.(*Fs).Put
       8MB  0.21% 96.47%    26.79MB  0.69%  github.com/rclone/rclone/vfs.newRWFileHandle
    4.04MB   0.1% 96.57%    21.04MB  0.54%  github.com/rclone/rclone/vfs.(*Dir)._readDirFromEntries
       2MB 0.051% 96.62%   100.30MB  2.58%  github.com/rclone/rclone/cmd/mount.(*Dir).Create
    1.50MB 0.039% 96.66%    26.22MB  0.67%  github.com/Azure/azure-storage-blob-go/azblob.(*SharedKeyCredential).New.func1
       1MB 0.026% 96.69%    30.01MB  0.77%  github.com/rclone/rclone/vfs/vfscache.(*Cache)._get
    0.50MB 0.013% 96.70%    24.53MB  0.63%  github.com/rclone/rclone/cmd/mount.(*Dir).Lookup
         0     0% 96.70%   137.96MB  3.55%  bazil.org/fuse/fs.(*Server).Serve.func1
         0     0% 96.70%   137.96MB  3.55%  bazil.org/fuse/fs.(*Server).handleRequest
         0     0% 96.70%   137.96MB  3.55%  bazil.org/fuse/fs.(*Server).serve
         0     0% 96.70%    23.71MB  0.61%  context.WithTimeout
         0     0% 96.70%    50.43MB  1.30%  github.com/Azure/azure-pipeline-go/pipeline.(*pipeline).Do
         0     0% 96.70%    50.43MB  1.30%  github.com/Azure/azure-pipeline-go/pipeline.PolicyFunc.Do
         0     0% 96.70%  3606.21MB 92.79%  github.com/Azure/azure-storage-blob-go/azblob.(*copier).sendChunk
         0     0% 96.70%    28.50MB  0.73%  github.com/Azure/azure-storage-blob-go/azblob.BlobURL.GetProperties
         0     0% 96.70%    50.43MB  1.30%  github.com/Azure/azure-storage-blob-go/azblob.NewRetryPolicyFactory.func1.1
         0     0% 96.70%    50.43MB  1.30%  github.com/Azure/azure-storage-blob-go/azblob.NewTelemetryPolicyFactory.func1.1
         0     0% 96.70%    50.43MB  1.30%  github.com/Azure/azure-storage-blob-go/azblob.NewUniqueRequestIDPolicyFactory.func1.1
         0     0% 96.70%  3611.92MB 92.94%  github.com/Azure/azure-storage-blob-go/azblob.UploadStreamToBlockBlob
         0     0% 96.70%    28.50MB  0.73%  github.com/Azure/azure-storage-blob-go/azblob.blobClient.GetProperties
         0     0% 96.70%  3611.92MB 92.94%  github.com/Azure/azure-storage-blob-go/azblob.copyFromReader
         0     0% 96.70%    24.71MB  0.64%  github.com/Azure/azure-storage-blob-go/azblob.responderPolicy.Do
         0     0% 96.70%       20MB  0.51%  github.com/rclone/rclone/backend/azureblob.(*Fs).list
         0     0% 96.70%  3686.24MB 94.85%  github.com/rclone/rclone/backend/azureblob.(*Object).Update
         0     0% 96.70%  3611.92MB 92.94%  github.com/rclone/rclone/backend/azureblob.(*Object).Update.func1
         0     0% 96.70%    46.01MB  1.18%  github.com/rclone/rclone/backend/azureblob.(*Object).decodeMetaDataFromPropertiesResponse
         0     0% 96.70%    74.51MB  1.92%  github.com/rclone/rclone/backend/azureblob.(*Object).readMetaData
         0     0% 96.70%    28.50MB  0.73%  github.com/rclone/rclone/backend/azureblob.(*Object).readMetaData.func1
         0     0% 96.70%  3652.22MB 93.97%  github.com/rclone/rclone/fs.pacerInvoker
         0     0% 96.70%  3702.75MB 95.27%  github.com/rclone/rclone/fs/operations.Copy
         0     0% 96.70%       41MB  1.06%  github.com/rclone/rclone/lib/pacer.(*Pacer).Call
         0     0% 96.70%  3611.21MB 92.92%  github.com/rclone/rclone/lib/pacer.(*Pacer).CallNoRetry (inline)
         0     0% 96.70%  3652.22MB 93.97%  github.com/rclone/rclone/lib/pacer.(*Pacer).call
         0     0% 96.70%    21.50MB  0.55%  github.com/rclone/rclone/vfs.(*Dir).Create
         0     0% 96.70%    24.03MB  0.62%  github.com/rclone/rclone/vfs.(*Dir).Stat
         0     0% 96.70%    29.54MB  0.76%  github.com/rclone/rclone/vfs.(*Dir)._readDir
         0     0% 96.70%    24.03MB  0.62%  github.com/rclone/rclone/vfs.(*Dir).stat
         0     0% 96.70%    76.80MB  1.98%  github.com/rclone/rclone/vfs.(*File).Open
         0     0% 96.70%       20MB  0.51%  github.com/rclone/rclone/vfs.(*File).Path
         0     0% 96.70%    26.79MB  0.69%  github.com/rclone/rclone/vfs.(*File).openRW
         0     0% 96.70%    30.01MB  0.77%  github.com/rclone/rclone/vfs/vfscache.(*Cache).Exists
         0     0% 96.70%    30.01MB  0.77%  github.com/rclone/rclone/vfs/vfscache.(*Cache).get
         0     0% 96.70%  3694.24MB 95.05%  github.com/rclone/rclone/vfs/vfscache.(*Item).Close.func2
         0     0% 96.70%  3694.24MB 95.05%  github.com/rclone/rclone/vfs/vfscache.(*Item)._store
         0     0% 96.70%  3694.24MB 95.05%  github.com/rclone/rclone/vfs/vfscache.(*Item).store
         0     0% 96.70%  3694.24MB 95.05%  github.com/rclone/rclone/vfs/vfscache/writeback.(*WriteBack).upload
         0     0% 96.70%       20MB  0.51%  path.Join
         0     0% 96.70%       20MB  0.51%  strings.(*Builder).Grow (inline)
         0     0% 96.70%       20MB  0.51%  strings.Join
         0     0% 96.70%  3613.04MB 92.96%  sync.(*Pool).Get

Best,
Raymond

Hmm, the leak is in the Azure blob SDK.

I have a feeling that this is fixed in the SDK version in use in the beta.

Was that from the beta?

Maybe I forgot to merge something.

I used rclone version:
rclone v1.54.0-beta.5040.71edc75ca

I dug into this again and remembered I made an issue about this on the azure blob SDK

@Raimondas Until the bug @ncw mentioned in the Azure blob SDK is fixed I suggest you stick with rclone 1.52.3, that version works fine with low memory usage (can't say for sure about CPU but I think the high usage might be related to garbage collection).

See my original post about the problem here, I think it is the same as yours: Rclone 1.53.3 memory (leak?) problem

Hi Daniel,
You saved the day! Thanks a lot for your advice!
I installed rclone v1.52.3 and after 1h45m I see rclone has eaten only 450Mb. That's success!

Saved profile in /root/pprof/pprof.rclone.alloc_objects.alloc_space.inuse_objects.inuse_space.035.pb.gz
File: rclone
Type: inuse_space
Time: Jan 9, 2021 at 12:33pm (EST)
Showing nodes accounting for 138.46MB, 98.19% of 141.01MB total
Dropped 25 nodes (cum <= 0.71MB)
      flat  flat%   sum%        cum   cum%
   20.50MB 14.54% 14.54%    20.50MB 14.54%  strings.(*Builder).grow (inline)
      18MB 12.77% 27.31%       18MB 12.77%  github.com/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.BlobGetPropertiesResponse.NewMetadata
   13.50MB  9.58% 36.88%    13.50MB  9.58%  github.com/rclone/rclone/vfs.newFile
      12MB  8.51% 45.40%       12MB  8.51%  github.com/rclone/rclone/vfs.newDir
    8.81MB  6.24% 51.64%     8.81MB  6.24%  github.com/rclone/rclone/vendor/bazil.org/fuse/fs.(*Server).saveNode
       7MB  4.96% 56.60%    46.01MB 32.63%  github.com/rclone/rclone/backend/azureblob.(*Fs).Put
       6MB  4.26% 60.86%        6MB  4.26%  time.AfterFunc
       6MB  4.26% 65.12%        6MB  4.26%  net/textproto.(*Reader).ReadMIMEHeader
    5.50MB  3.90% 69.02%    11.50MB  8.16%  context.WithDeadline
    5.26MB  3.73% 72.74%     5.26MB  3.73%  github.com/rclone/rclone/vfs.(*Dir).addObject
    4.54MB  3.22% 75.96%    17.54MB 12.44%  github.com/rclone/rclone/vfs.(*Dir)._readDirFromEntries
    4.50MB  3.19% 79.16%     4.50MB  3.19%  os.newFile
       4MB  2.84% 81.99%        4MB  2.84%  github.com/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.uploadStream.func1
       4MB  2.84% 84.83%        4MB  2.84%  context.(*cancelCtx).Done
       4MB  2.84% 87.67%        4MB  2.84%  encoding/xml.copyValue
    3.50MB  2.48% 90.15%    27.76MB 19.69%  github.com/rclone/rclone/vfs.newRWFileHandle
       3MB  2.13% 92.28%        3MB  2.13%  github.com/rclone/rclone/fs.NewDir (inline)
    2.50MB  1.77% 94.05%     3.64MB  2.58%  github.com/rclone/rclone/vendor/bazil.org/fuse.(*Conn).ReadRequest
    1.50MB  1.06% 95.11%     1.50MB  1.06%  github.com/rclone/rclone/vendor/github.com/aws/aws-sdk-go/aws/endpoints.init
    1.50MB  1.06% 96.18%     1.50MB  1.06%  github.com/rclone/rclone/vfs.(*File).addWriter
    1.20MB  0.85% 97.03%     1.20MB  0.85%  time.startTimer
    1.14MB  0.81% 97.84%     1.14MB  0.81%  github.com/rclone/rclone/vendor/bazil.org/fuse.allocMessage
    0.50MB  0.35% 98.19%     1.10MB  0.78%  time.NewTimer
         0     0% 98.19%    11.50MB  8.16%  context.WithTimeout
         0     0% 98.19%     1.05MB  0.74%  crypto/tls.(*Conn).Handshake
         0     0% 98.19%     1.05MB  0.74%  crypto/tls.(*Conn).clientHandshake
         0     0% 98.19%     1.05MB  0.74%  crypto/tls.(*clientHandshakeState).doFullHandshake
         0     0% 98.19%     1.05MB  0.74%  crypto/tls.(*clientHandshakeState).handshake
         0     0% 98.19%        4MB  2.84%  encoding/xml.(*Decoder).Decode (inline)
         0     0% 98.19%        4MB  2.84%  encoding/xml.(*Decoder).DecodeElement
         0     0% 98.19%        4MB  2.84%  encoding/xml.(*Decoder).unmarshal
         0     0% 98.19%        4MB  2.84%  encoding/xml.(*Decoder).unmarshalPath
         0     0% 98.19%        4MB  2.84%  encoding/xml.Unmarshal
         0     0% 98.19%        7MB  4.96%  github.com/rclone/rclone/backend/azureblob.(*Fs).List
         0     0% 98.19%        3MB  2.13%  github.com/rclone/rclone/backend/azureblob.(*Fs).itemToDirEntry
         0     0% 98.19%        7MB  4.96%  github.com/rclone/rclone/backend/azureblob.(*Fs).list
         0     0% 98.19%        4MB  2.84%  github.com/rclone/rclone/backend/azureblob.(*Fs).list.func1
         0     0% 98.19%        7MB  4.96%  github.com/rclone/rclone/backend/azureblob.(*Fs).listDir
         0     0% 98.19%        3MB  2.13%  github.com/rclone/rclone/backend/azureblob.(*Fs).listDir.func1
         0     0% 98.19%    39.01MB 27.66%  github.com/rclone/rclone/backend/azureblob.(*Object).Update
         0     0% 98.19%        5MB  3.55%  github.com/rclone/rclone/backend/azureblob.(*Object).Update.func1
         0     0% 98.19%    18.50MB 13.12%  github.com/rclone/rclone/backend/azureblob.(*Object).decodeMetaDataFromPropertiesResponse
         0     0% 98.19%    34.01MB 24.12%  github.com/rclone/rclone/backend/azureblob.(*Object).readMetaData
         0     0% 98.19%    16.10MB 11.42%  github.com/rclone/rclone/backend/azureblob.(*Object).readMetaData.func1
         0     0% 98.19%     5.10MB  3.62%  github.com/rclone/rclone/backend/azureblob.httpClientFactory.func1.1
         0     0% 98.19%    40.26MB 28.55%  github.com/rclone/rclone/cmd/mount.(*Dir).Create
         0     0% 98.19%    19.54MB 13.86%  github.com/rclone/rclone/cmd/mount.(*Dir).Lookup
         0     0% 98.19%        1MB  0.71%  github.com/rclone/rclone/cmd/mount.(*Dir).ReadDirAll
         0     0% 98.19%    49.51MB 35.11%  github.com/rclone/rclone/cmd/mount.(*FileHandle).Flush
         0     0% 98.19%     3.64MB  2.58%  github.com/rclone/rclone/cmd/mount.mount.func1
         0     0% 98.19%    25.10MB 17.80%  github.com/rclone/rclone/fs.pacerInvoker
         0     0% 98.19%     5.10MB  3.62%  github.com/rclone/rclone/fs/fshttp.(*Transport).RoundTrip
         0     0% 98.19%        3MB  2.13%  github.com/rclone/rclone/fs/list.DirSorted
         0     0% 98.19%    46.01MB 32.63%  github.com/rclone/rclone/fs/operations.Copy
         0     0% 98.19%    20.10MB 14.26%  github.com/rclone/rclone/lib/pacer.(*Pacer).Call
         0     0% 98.19%        5MB  3.55%  github.com/rclone/rclone/lib/pacer.(*Pacer).CallNoRetry (inline)
         0     0% 98.19%    25.10MB 17.80%  github.com/rclone/rclone/lib/pacer.(*Pacer).call
         0     0% 98.19%     1.14MB  0.81%  github.com/rclone/rclone/vendor/bazil.org/fuse.getMessage (inline)
         0     0% 98.19%     3.64MB  2.58%  github.com/rclone/rclone/vendor/bazil.org/fuse/fs.(*Server).Serve
         0     0% 98.19%   119.12MB 84.48%  github.com/rclone/rclone/vendor/bazil.org/fuse/fs.(*Server).Serve.func1
         0     0% 98.19%   119.12MB 84.48%  github.com/rclone/rclone/vendor/bazil.org/fuse/fs.(*Server).handleRequest
         0     0% 98.19%     8.81MB  6.24%  github.com/rclone/rclone/vendor/bazil.org/fuse/fs.(*Server).saveLookup
         0     0% 98.19%   119.12MB 84.48%  github.com/rclone/rclone/vendor/bazil.org/fuse/fs.(*Server).serve
         0     0% 98.19%    20.60MB 14.61%  github.com/rclone/rclone/vendor/github.com/Azure/azure-pipeline-go/pipeline.(*pipeline).Do
         0     0% 98.19%    20.60MB 14.61%  github.com/rclone/rclone/vendor/github.com/Azure/azure-pipeline-go/pipeline.PolicyFunc.Do
         0     0% 98.19%     9.10MB  6.45%  github.com/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.(*SharedKeyCredential).New.func1
         0     0% 98.19%    16.10MB 11.42%  github.com/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.BlobURL.GetProperties
         0     0% 98.19%        4MB  2.84%  github.com/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.ContainerURL.ListBlobsHierarchySegment
         0     0% 98.19%     5.10MB  3.62%  github.com/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.NewRequestLogPolicyFactory.func1.1
         0     0% 98.19%    20.60MB 14.61%  github.com/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.NewRetryPolicyFactory.func1.1
         0     0% 98.19%    20.60MB 14.61%  github.com/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.NewTelemetryPolicyFactory.func1.1
         0     0% 98.19%    20.60MB 14.61%  github.com/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.NewUniqueRequestIDPolicyFactory.func1.1
         0     0% 98.19%        5MB  3.55%  github.com/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.UploadStreamToBlockBlob
         0     0% 98.19%    16.10MB 11.42%  github.com/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.blobClient.GetProperties
         0     0% 98.19%        4MB  2.84%  github.com/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.containerClient.ListBlobHierarchySegment
         0     0% 98.19%        4MB  2.84%  github.com/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.containerClient.listBlobHierarchySegmentResponder
         0     0% 98.19%     9.10MB  6.45%  github.com/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.responderPolicy.Do
         0     0% 98.19%        5MB  3.55%  github.com/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.uploadStream
         0     0% 98.19%    12.50MB  8.87%  github.com/rclone/rclone/vfs.(*Dir).Create
         0     0% 98.19%        1MB  0.71%  github.com/rclone/rclone/vfs.(*Dir).ReadDirAll
         0     0% 98.19%    19.54MB 13.86%  github.com/rclone/rclone/vfs.(*Dir).Stat
         0     0% 98.19%    20.54MB 14.57%  github.com/rclone/rclone/vfs.(*Dir)._readDir
         0     0% 98.19%    19.54MB 13.86%  github.com/rclone/rclone/vfs.(*Dir).stat
         0     0% 98.19%    27.76MB 19.69%  github.com/rclone/rclone/vfs.(*File).Open
         0     0% 98.19%        9MB  6.38%  github.com/rclone/rclone/vfs.(*File).Path
         0     0% 98.19%    27.76MB 19.69%  github.com/rclone/rclone/vfs.(*File).openRW
         0     0% 98.19%    11.50MB  8.16%  github.com/rclone/rclone/vfs.(*File).osPath
         0     0% 98.19%    49.51MB 35.11%  github.com/rclone/rclone/vfs.(*RWFileHandle).Flush
         0     0% 98.19%    49.51MB 35.11%  github.com/rclone/rclone/vfs.(*RWFileHandle).flushWrites
         0     0% 98.19%    21.26MB 15.08%  github.com/rclone/rclone/vfs.(*RWFileHandle).openPending
         0     0% 98.19%    42.01MB 29.79%  github.com/rclone/rclone/vfs/vfscache.(*Cache).Store
         0     0% 98.19%    11.50MB  8.16%  github.com/rclone/rclone/vfs/vfscache.(*Cache).ToOSPath (inline)
         0     0% 98.19%    42.01MB 29.79%  github.com/rclone/rclone/vfs/vfscache.copyObj
         0     0% 98.19%     5.10MB  3.62%  net/http.(*Client).Do (inline)
         0     0% 98.19%     5.10MB  3.62%  net/http.(*Client).do
         0     0% 98.19%     5.10MB  3.62%  net/http.(*Client).send
         0     0% 98.19%     5.10MB  3.62%  net/http.(*Transport).RoundTrip (inline)
         0     0% 98.19%     5.10MB  3.62%  net/http.(*Transport).roundTrip
         0     0% 98.19%     1.05MB  0.74%  net/http.(*persistConn).addTLS.func2
         0     0% 98.19%        6MB  4.26%  net/http.(*persistConn).readLoop
         0     0% 98.19%        6MB  4.26%  net/http.(*persistConn).readResponse
         0     0% 98.19%     1.10MB  0.78%  net/http.(*persistConn).roundTrip
         0     0% 98.19%        6MB  4.26%  net/http.ReadResponse
         0     0% 98.19%     5.10MB  3.62%  net/http.send
         0     0% 98.19%     4.50MB  3.19%  os.OpenFile
         0     0% 98.19%     4.50MB  3.19%  os.openFileNolog
         0     0% 98.19%        9MB  6.38%  path.Join
         0     0% 98.19%    11.50MB  8.16%  path/filepath.Join (inline)
         0     0% 98.19%    11.50MB  8.16%  path/filepath.join
         0     0% 98.19%     1.50MB  1.06%  runtime.doInit
         0     0% 98.19%     1.50MB  1.06%  runtime.main
         0     0% 98.19%    20.50MB 14.54%  strings.(*Builder).Grow (inline)
         0     0% 98.19%    20.50MB 14.54%  strings.Join
         0     0% 98.19%     1.05MB  0.74%  sync.(*Once).Do (inline)
         0     0% 98.19%     1.05MB  0.74%  sync.(*Once).doSlow
         0     0% 98.19%     1.14MB  0.81%  sync.(*Pool).Get

Best,
Raymond

update.
rclone v1.52.3

rclone mount --log-file=/var/log/dcmsys/rclone.log --log-level ERROR --config=/etc/rclone.conf --allow-other --default-permissions --max-read-ahead=4M --vfs-cache-mode=writes --vfs-cache-max-size=1G --azureblob-memory-pool-flush-time=1m0s --buffer-size=0 --azureblob-memory-pool-use-mmap --use-mmap --dir-cache-time=2h --timeout=1h --cache-dir=/var/dcmsys/cache --uid=1100 --gid=1100 --cache-chunk-no-memory --rc --size-only --allow-non-empty blob:studiesh /var/dcmsys/storages/GCP

after 2 days of running, htop shows that rclone uses about 9Gb or ram:

1860391 root 20 0 9.8G 9210M 10612 S 0.0 28.6 18h41:38 rclone mount --log-file=/var/log/dcmsys/rclone.log --log-level ERROR --config=/etc/rcl

Here is the memory profile:
> Fetching profile over HTTP from localhost:5572/debug/pprof/heap
> Saved profile in /root/pprof/pprof.rclone.alloc_objects.alloc_space.inuse_objects.inuse_space.037.pb.gz
> File: rclone
> Type: inuse_space
> Time: Jan 11, 2021 at 9:10am (EST)
> Showing nodes accounting for 4060.02MB, 97.81% of 4151MB total
> Dropped 99 nodes (cum <= 20.75MB)
>       flat  flat%   sum%        cum   cum%
>   712.17MB 17.16% 17.16%   712.17MB 17.16%  github.c/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.BlobGetPropertiesResponse.NewMetadata
>   674.10MB 16.24% 33.40%   674.10MB 16.24%  strings.(*Builder).grow (inline)
>   521.61MB 12.57% 45.96%   521.61MB 12.57%  github.c/rclone/rclone/vfs.newFile
>   270.02MB  6.50% 52.47%   487.07MB 11.73%  context.WithDeadline
>   258.01MB  6.22% 58.68%   258.01MB  6.22%  net/textproto.(*Reader).ReadMIMEHeader
>   245.52MB  5.91% 64.60%   245.52MB  5.91%  context.(*cancelCtx).Done
>   242.03MB  5.83% 70.43%  1852.36MB 44.62%  github.c/rclone/rclone/backend/azureblob.(*Fs).Put
>   232.52MB  5.60% 76.03%   232.52MB  5.60%  os.newFile
>   180.51MB  4.35% 80.38%   198.55MB  4.78%  time.AfterFunc
>   161.33MB  3.89% 84.26%   161.33MB  3.89%  github.c/rclone/rclone/vfs.(*Dir).addObject
>   146.51MB  3.53% 87.79%   964.89MB 23.24%  github.c/rclone/rclone/vfs.newRWFileHandle
>   141.01MB  3.40% 91.19%   153.53MB  3.70%  github.c/rclone/rclone/vendor/bazil.org/fuse.(*Conn).ReadRequest
>   128.04MB  3.08% 94.28%   128.04MB  3.08%  github.c/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.uploadStream.func1
>       64MB  1.54% 95.82%       64MB  1.54%  encoding/base64.(*Encoding).EncodeToString
>    40.50MB  0.98% 96.79%    40.50MB  0.98%  github.c/rclone/rclone/vfs.(*File).addWriter
>    23.58MB  0.57% 97.36%    23.58MB  0.57%  time.startTimer
>    11.50MB  0.28% 97.64%  1496.51MB 36.05%  github.c/rclone/rclone/cmd/mount.(*Dir).Create
>     7.06MB  0.17% 97.81%    25.06MB   0.6%  github.c/rclone/rclone/vfs.(*Dir)._readDirFromEntries
>          0     0% 97.81%   487.07MB 11.73%  context.WithTimeout
>          0     0% 97.81%       21MB  0.51%  encoding/xml.(*Decoder).Decode (inline)
>          0     0% 97.81%       21MB  0.51%  encoding/xml.(*Decoder).DecodeElement
>          0     0% 97.81%       21MB  0.51%  encoding/xml.(*Decoder).unmarshal
>          0     0% 97.81%       21MB  0.51%  encoding/xml.(*Decoder).unmarshalPath
>          0     0% 97.81%       21MB  0.51%  encoding/xml.Unmarshal
>          0     0% 97.81%       27MB  0.65%  github.c/rclone/rclone/backend/azureblob.(*Fs).List
>          0     0% 97.81%       37MB  0.89%  github.c/rclone/rclone/backend/azureblob.(*Fs).list
>          0     0% 97.81%    27.50MB  0.66%  github.c/rclone/rclone/backend/azureblob.(*Fs).list.func1
>          0     0% 97.81%       27MB  0.65%  github.c/rclone/rclone/backend/azureblob.(*Fs).listDir
>          0     0% 97.81%  1623.84MB 39.12%  github.c/rclone/rclone/backend/azureblob.(*Object).Update
>          0     0% 97.81%   224.08MB  5.40%  github.c/rclone/rclone/backend/azureblob.(*Object).Update.func1
>          0     0% 97.81%   774.68MB 18.66%  github.c/rclone/rclone/backend/azureblob.(*Object).decodeMetaDataFromPropertiesResponse
>          0     0% 97.81%  1399.23MB 33.71%  github.c/rclone/rclone/backend/azureblob.(*Object).readMetaData
>          0     0% 97.81%   624.05MB 15.03%  github.c/rclone/rclone/backend/azureblob.(*Object).readMetaData.func1
>          0     0% 97.81%   228.52MB  5.51%  github.c/rclone/rclone/backend/azureblob.httpClientFactory.func1.1
>          0     0% 97.81%    35.56MB  0.86%  github.c/rclone/rclone/cmd/mount.(*Dir).Lookup
>          0     0% 97.81%  1923.88MB 46.35%  github.c/rclone/rclone/cmd/mount.(*FileHandle).Flush
>          0     0% 97.81%   153.53MB  3.70%  github.c/rclone/rclone/cmd/mount.mount.func1
>          0     0% 97.81%   876.14MB 21.11%  github.c/rclone/rclone/fs.pacerInvoker
>          0     0% 97.81%   228.52MB  5.51%  github.c/rclone/rclone/fs/fshttp.(*Transport).RoundTrip
>          0     0% 97.81%  1853.87MB 44.66%  github.c/rclone/rclone/fs/operations.Copy
>          0     0% 97.81%   651.56MB 15.70%  github.c/rclone/rclone/lib/pacer.(*Pacer).Call
>          0     0% 97.81%   223.58MB  5.39%  github.c/rclone/rclone/lib/pacer.(*Pacer).CallNoRetry (inline)
>          0     0% 97.81%   876.14MB 21.11%  github.c/rclone/rclone/lib/pacer.(*Pacer).call
>          0     0% 97.81%   153.53MB  3.70%  github.c/rclone/rclone/vendor/bazil.org/fuse/fs.(*Server).Serve
>          0     0% 97.81%  3459.25MB 83.34%  github.c/rclone/rclone/vendor/bazil.org/fuse/fs.(*Server).Serve.func1
>          0     0% 97.81%  3477.28MB 83.77%  github.c/rclone/rclone/vendor/bazil.org/fuse/fs.(*Server).handleRequest
>          0     0% 97.81%  3477.28MB 83.77%  github.c/rclone/rclone/vendor/bazil.org/fuse/fs.(*Server).serve
>          0     0% 97.81%   737.59MB 17.77%  github.c/rclone/rclone/vendor/github.com/Azure/azure-pipeline-go/pipeline.(*pipeline).Do
>          0     0% 97.81%   737.59MB 17.77%  github.c/rclone/rclone/vendor/github.c/Azure/azure-pipeline-go/pipeline.PolicyFunc.Do
>          0     0% 97.81%   250.02MB  6.02%  github.c/rclone/rclone/vendor/github.c/Azure/azure-storage-blob-go/azblob.(*SharedKeyCredential).New.func1
>          0     0% 97.81%    83.54MB  2.01%  github.c/rclone/rclone/vendor/github.c/Azure/azure-storage-blob-go/azblob.(*uploadStreamToBlockBlobOptions).end
>          0     0% 97.81%   624.05MB 15.03%  github.c/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.BlobURL.GetProperties
>          0     0% 97.81%    83.54MB  2.01%  github.c/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.BlockBlobURL.Upload
>          0     0% 97.81%    27.50MB  0.66%  github.c/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.ContainerURL.ListBlobsHierarchySegment
>          0     0% 97.81%   228.52MB  5.51%  github.c/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.NewRequestLogPolicyFactory.func1.1
>          0     0% 97.81%   737.59MB 17.77%  github.c/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.NewRetryPolicyFactory.func1.1
>          0     0% 97.81%   737.59MB 17.77%  github.c/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.NewTelemetryPolicyFactory.func1.1
>          0     0% 97.81%   737.59MB 17.77%  github./rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.NewUniqueRequestIDPolicyFactory.func1.1
>          0     0% 97.81%   224.08MB  5.40%  github./rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.UploadStreamToBlockBlob
>          0     0% 97.81%   624.05MB 15.03%  github./rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.blobClient.GetProperties
>          0     0% 97.81%    83.54MB  2.01%  github./rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.blockBlobClient.Upload
>          0     0% 97.81%    27.50MB  0.66%  github./rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.containerClient.ListBlobHierarchySegment
>          0     0% 97.81%       21MB  0.51%  github./rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.containerClient.listBlobHierarchySegmentResponder
>          0     0% 97.81%   249.52MB  6.01%  github./rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.responderPolicy.Do
>          0     0% 97.81%   224.08MB  5.40%  github./rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.uploadStream
>          0     0% 97.81%   520.11MB 12.53%  github./rclone/rclone/vfs.(*Dir).Create
>          0     0% 97.81%    35.56MB  0.86%  github./rclone/rclone/vfs.(*Dir).Stat
>          0     0% 97.81%    39.56MB  0.95%  github.c/rclone/rclone/vfs.(*Dir)._readDir
>          0     0% 97.81%    35.56MB  0.86%  github./rclone/rclone/vfs.(*Dir).stat
>          0     0% 97.81%   964.89MB 23.24%  github./rclone/rclone/vfs.(*File).Open
>          0     0% 97.81%   282.03MB  6.79%  github./rclone/rclone/vfs.(*File).Path
>          0     0% 97.81%   964.89MB 23.24%  github./rclone/rclone/vfs.(*File).openRW
>          0     0% 97.81%   389.56MB  9.38%  github/rclone/rclone/vfs.(*File).osPath
>          0     0% 97.81%  1923.88MB 46.35%  github./rclone/rclone/vfs.(*RWFileHandle).Flush
>          0     0% 97.81%  1923.88MB 46.35%  github.c/rclone/rclone/vfs.(*RWFileHandle).flushWrites
>          0     0% 97.81%   781.89MB 18.84%  github/rclone/rclone/vfs.(*RWFileHandle).openPending
>          0     0% 97.81%  1641.85MB 39.55%  github.c/rclone/rclone/vfs/vfscache.(*Cache).Store
>          0     0% 97.81%   389.56MB  9.38%  github.c/rclone/rclone/vfs/vfscache.(*Cache).ToOSPath (inline)
>          0     0% 97.81%  1641.85MB 39.55%  github.c/rclone/rclone/vfs/vfscache.copyObj
>          0     0% 97.81%   228.52MB  5.51%  net/http.(*Client).Do (inline)
>          0     0% 97.81%   228.52MB  5.51%  net/http.(*Client).do
>          0     0% 97.81%   228.52MB  5.51%  net/http.(*Client).send
>          0     0% 97.81%   228.52MB  5.51%  net/http.(*Transport).RoundTrip (inline)
>          0     0% 97.81%   228.52MB  5.51%  net/http.(*Transport).roundTrip
>          0     0% 97.81%   258.01MB  6.22%  net/http.(*persistConn).readLoop
>          0     0% 97.81%   258.01MB  6.22%  net/http.(*persistConn).readResponse
>          0     0% 97.81%   258.01MB  6.22%  net/http.ReadResponse
>          0     0% 97.81%   228.52MB  5.51%  net/http.send
>          0     0% 97.81%   232.52MB  5.60%  os.OpenFile
>          0     0% 97.81%   232.52MB  5.60%  os.openFileNolog
>          0     0% 97.81%   284.03MB  6.84%  path.Join
>          0     0% 97.81%   389.56MB  9.38%  path/filepath.Join (inline)
>          0     0% 97.81%   389.56MB  9.38%  path/filepath.join
>          0     0% 97.81%   674.10MB 16.24%  strings.(*Builder).Grow (inline)
>          0     0% 97.81%   673.60MB 16.23%  strings.Join

Best,
Raymond

Would you mind trying this beta or later?

This has had the large file upload code reworked to use the facilities from the SDK entirely. I think this may help, but I'm not 100% sure!

Uptime 3h:

rclone v1.54.0-beta.5060.e0b5a13a1

  • os/arch: linux/amd64
  • go version: go1.15.6

htop:
2703199 root 20 0 10.6G 9745M 31376 S 24.6 30.3 40:39.95 rclone mount --log-file=/var/log/dcmsys/rclone.log --log-level ERROR --config=/etc/rclone.conf --allow-other --default-permissions --max-read-ahead=4M --vfs-cache-mode=writes --vfs-cache-max-size=1G --azureblob-memory-pool-f

Saved profile in /root/pprof/pprof.rclone.alloc_objects.alloc_space.inuse_objects.inuse_space.041.pb.gz
File: rclone
Type: inuse_space
Time: Jan 13, 2021 at 4:15am (EST)
Showing nodes accounting for 4827.99MB, 96.60% of 4997.81MB total
Dropped 147 nodes (cum <= 24.99MB)
      flat  flat%   sum%        cum   cum%
 4601.54MB 92.07% 92.07%  4601.54MB 92.07%  github.com/Azure/azure-storage-blob-go/azblob.copyFromReader.func1
   58.01MB  1.16% 93.23%    58.01MB  1.16%  github.com/Azure/azure-storage-blob-go/azblob.BlobGetPropertiesResponse.NewMetadata
   36.01MB  0.72% 93.95%    46.01MB  0.92%  github.com/rclone/rclone/vfs/vfscache.newItem
   34.01MB  0.68% 94.63%    34.01MB  0.68%  github.com/rclone/rclone/vfs.newFile
   27.33MB  0.55% 95.18%    27.33MB  0.55%  bazil.org/fuse.allocMessage
   22.50MB  0.45% 95.63%    31.77MB  0.64%  context.WithDeadline
   15.50MB  0.31% 95.94%  4735.57MB 94.75%  github.com/rclone/rclone/backend/azureblob.(*Fs).Put
   13.50MB  0.27% 96.21%    26.74MB  0.53%  github.com/rclone/rclone/vfs.newRWFileHandle
    9.50MB  0.19% 96.40%    36.83MB  0.74%  bazil.org/fuse.(*Conn).ReadRequest
    5.09MB   0.1% 96.50%    27.09MB  0.54%  github.com/rclone/rclone/vfs.(*Dir)._readDirFromEntries
       2MB  0.04% 96.54%    38.55MB  0.77%  github.com/Azure/azure-storage-blob-go/azblob.(*SharedKeyCredential).New.func1
    1.50MB  0.03% 96.57%   129.75MB  2.60%  github.com/rclone/rclone/cmd/mount.(*Dir).Create
    0.50MB  0.01% 96.58%  4610.32MB 92.25%  github.com/Azure/azure-storage-blob-go/azblob.copyFromReader
    0.50MB  0.01% 96.59%  4610.82MB 92.26%  github.com/Azure/azure-storage-blob-go/azblob.UploadStreamToBlockBlob
    0.50MB  0.01% 96.60%    70.82MB  1.42%  github.com/Azure/azure-storage-blob-go/azblob.NewRetryPolicyFactory.func1.1
         0     0% 96.60%    27.33MB  0.55%  bazil.org/fuse.getMessage (inline)
         0     0% 96.60%    36.83MB  0.74%  bazil.org/fuse/fs.(*Server).Serve
         0     0% 96.60%   176.84MB  3.54%  bazil.org/fuse/fs.(*Server).Serve.func1
         0     0% 96.60%   176.84MB  3.54%  bazil.org/fuse/fs.(*Server).handleRequest
         0     0% 96.60%   176.84MB  3.54%  bazil.org/fuse/fs.(*Server).serve
         0     0% 96.60%    31.77MB  0.64%  context.WithTimeout
         0     0% 96.60%    70.82MB  1.42%  github.com/Azure/azure-pipeline-go/pipeline.(*pipeline).Do
         0     0% 96.60%    70.82MB  1.42%  github.com/Azure/azure-pipeline-go/pipeline.PolicyFunc.Do
         0     0% 96.60%  4602.04MB 92.08%  github.com/Azure/azure-storage-blob-go/azblob.(*copier).sendChunk
         0     0% 96.60%    47.50MB  0.95%  github.com/Azure/azure-storage-blob-go/azblob.BlobURL.GetProperties
         0     0% 96.60%    25.54MB  0.51%  github.com/Azure/azure-storage-blob-go/azblob.NewRequestLogPolicyFactory.func1.1
         0     0% 96.60%    70.82MB  1.42%  github.com/Azure/azure-storage-blob-go/azblob.NewTelemetryPolicyFactory.func1.1
         0     0% 96.60%    70.82MB  1.42%  github.com/Azure/azure-storage-blob-go/azblob.NewUniqueRequestIDPolicyFactory.func1.1
         0     0% 96.60%    47.50MB  0.95%  github.com/Azure/azure-storage-blob-go/azblob.blobClient.GetProperties
         0     0% 96.60%    36.04MB  0.72%  github.com/Azure/azure-storage-blob-go/azblob.responderPolicy.Do
         0     0% 96.60%  4722.57MB 94.49%  github.com/rclone/rclone/backend/azureblob.(*Object).Update
         0     0% 96.60%  4610.82MB 92.26%  github.com/rclone/rclone/backend/azureblob.(*Object).Update.func1
         0     0% 96.60%    64.51MB  1.29%  github.com/rclone/rclone/backend/azureblob.(*Object).decodeMetaDataFromPropertiesResponse
         0     0% 96.60%   112.02MB  2.24%  github.com/rclone/rclone/backend/azureblob.(*Object).readMetaData
         0     0% 96.60%    47.50MB  0.95%  github.com/rclone/rclone/backend/azureblob.(*Object).readMetaData.func1
         0     0% 96.60%    25.54MB  0.51%  github.com/rclone/rclone/backend/azureblob.httpClientFactory.func1.1
         0     0% 96.60%    35.86MB  0.72%  github.com/rclone/rclone/cmd/mount.(*Dir).Lookup
         0     0% 96.60%    36.83MB  0.74%  github.com/rclone/rclone/cmd/mount.mount.func2
         0     0% 96.60%  4669.32MB 93.43%  github.com/rclone/rclone/fs.pacerInvoker
         0     0% 96.60%  4738.57MB 94.81%  github.com/rclone/rclone/fs/operations.Copy
         0     0% 96.60%    58.78MB  1.18%  github.com/rclone/rclone/lib/pacer.(*Pacer).Call
         0     0% 96.60%  4609.55MB 92.23%  github.com/rclone/rclone/lib/pacer.(*Pacer).CallNoRetry (inline)
         0     0% 96.60%  4669.32MB 93.43%  github.com/rclone/rclone/lib/pacer.(*Pacer).call
         0     0% 96.60%    33.01MB  0.66%  github.com/rclone/rclone/vfs.(*Dir).Create
         0     0% 96.60%    35.86MB  0.72%  github.com/rclone/rclone/vfs.(*Dir).Stat
         0     0% 96.60%    37.86MB  0.76%  github.com/rclone/rclone/vfs.(*Dir)._readDir
         0     0% 96.60%    35.86MB  0.72%  github.com/rclone/rclone/vfs.(*Dir).stat
         0     0% 96.60%    95.75MB  1.92%  github.com/rclone/rclone/vfs.(*File).Open
         0     0% 96.60%    26.74MB  0.53%  github.com/rclone/rclone/vfs.(*File).openRW
         0     0% 96.60%    46.01MB  0.92%  github.com/rclone/rclone/vfs/vfscache.(*Cache).Exists
         0     0% 96.60%    46.01MB  0.92%  github.com/rclone/rclone/vfs/vfscache.(*Cache)._get
         0     0% 96.60%    46.01MB  0.92%  github.com/rclone/rclone/vfs/vfscache.(*Cache).get
         0     0% 96.60%  4720.57MB 94.45%  github.com/rclone/rclone/vfs/vfscache.(*Item).Close.func2
         0     0% 96.60%  4720.57MB 94.45%  github.com/rclone/rclone/vfs/vfscache.(*Item)._store
         0     0% 96.60%  4720.57MB 94.45%  github.com/rclone/rclone/vfs/vfscache.(*Item).store
         0     0% 96.60%  4720.57MB 94.45%  github.com/rclone/rclone/vfs/vfscache/writeback.(*WriteBack).upload
         0     0% 96.60%  4628.87MB 92.62%  sync.(*Pool).Get

for comparison:

rclone version
rclone v1.52.3

uptime 3 days

htop:
1860391 root 20 0 12.7G 12.0G 9228 S 25.4 38.1 29h42:23 rclone mount --log-file=/var/log/dcmsys/rclone.log --log-level ERROR --config=/etc/rclone.conf --allow-other --default-permissions --max-read-ahead=4M --vfs-cache-mode=writes --vfs-cache-max-size=1G --azureblob-memory-pool-f

Fetching profile over HTTP from http://localhost:5572/debug/pprof/heap
Saved profile in /root/pprof/pprof.rclone.alloc_objects.alloc_space.inuse_objects.inuse_space.038.pb.gz
File: rclone
Type: inuse_space
Time: Jan 13, 2021 at 1:21am (EST)
Showing nodes accounting for 5.62GB, 98.02% of 5.73GB total
Dropped 98 nodes (cum <= 0.03GB)
      flat  flat%   sum%        cum   cum%
    0.95GB 16.58% 16.58%     0.95GB 16.58%  github.com/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.BlobGetPropertiesResponse.NewMetadata
    0.90GB 15.62% 32.20%     0.90GB 15.62%  strings.(*Builder).grow (inline)
    0.68GB 11.91% 44.12%     0.68GB 11.91%  github.com/rclone/rclone/vfs.newFile
    0.37GB  6.40% 50.52%     0.65GB 11.29%  context.WithDeadline
    0.34GB  5.95% 56.46%     0.34GB  5.95%  net/textproto.(*Reader).ReadMIMEHeader
    0.33GB  5.81% 62.28%     0.33GB  5.81%  context.(*cancelCtx).Done
    0.31GB  5.45% 67.73%     0.31GB  5.45%  github.com/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.uploadStream.func1
    0.31GB  5.42% 73.15%     2.60GB 45.39%  github.com/rclone/rclone/backend/azureblob.(*Fs).Put
    0.31GB  5.36% 78.51%     0.31GB  5.36%  os.newFile
    0.25GB  4.39% 82.90%     0.26GB  4.51%  time.AfterFunc
    0.22GB  3.86% 86.75%     0.22GB  3.86%  github.com/rclone/rclone/vfs.(*Dir).addObject
    0.19GB  3.40% 90.15%     1.29GB 22.52%  github.com/rclone/rclone/vfs.newRWFileHandle
    0.19GB  3.38% 93.54%     0.25GB  4.42%  github.com/rclone/rclone/vendor/bazil.org/fuse.(*Conn).ReadRequest
    0.09GB  1.54% 95.08%     0.09GB  1.54%  encoding/base64.(*Encoding).EncodeToString
    0.06GB  1.04% 96.12%     0.06GB  1.04%  github.com/rclone/rclone/vendor/bazil.org/fuse.allocMessage
    0.06GB  1.02% 97.14%     0.06GB  1.02%  github.com/rclone/rclone/vfs.(*File).addWriter
    0.03GB  0.56% 97.70%     0.03GB  0.56%  time.startTimer
    0.02GB  0.32% 98.02%     1.99GB 34.70%  github.com/rclone/rclone/cmd/mount.(*Dir).Create
         0     0% 98.02%     0.65GB 11.29%  context.WithTimeout
         0     0% 98.02%     0.04GB  0.63%  encoding/xml.(*Decoder).Decode (inline)
         0     0% 98.02%     0.04GB  0.63%  encoding/xml.(*Decoder).DecodeElement
         0     0% 98.02%     0.04GB  0.63%  encoding/xml.(*Decoder).unmarshal
         0     0% 98.02%     0.04GB  0.63%  encoding/xml.(*Decoder).unmarshalPath
         0     0% 98.02%     0.04GB  0.63%  encoding/xml.Unmarshal
         0     0% 98.02%     0.03GB  0.57%  github.com/rclone/rclone/backend/azureblob.(*Fs).List
         0     0% 98.02%     0.05GB  0.95%  github.com/rclone/rclone/backend/azureblob.(*Fs).list
         0     0% 98.02%     0.04GB  0.76%  github.com/rclone/rclone/backend/azureblob.(*Fs).list.func1
         0     0% 98.02%     0.03GB   0.6%  github.com/rclone/rclone/backend/azureblob.(*Fs).listDir
         0     0% 98.02%     2.32GB 40.53%  github.com/rclone/rclone/backend/azureblob.(*Object).Update
         0     0% 98.02%     0.45GB  7.82%  github.com/rclone/rclone/backend/azureblob.(*Object).Update.func1
         0     0% 98.02%     1.04GB 18.08%  github.com/rclone/rclone/backend/azureblob.(*Object).decodeMetaDataFromPropertiesResponse
         0     0% 98.02%     1.88GB 32.83%  github.com/rclone/rclone/backend/azureblob.(*Object).readMetaData
         0     0% 98.02%     0.85GB 14.90%  github.com/rclone/rclone/backend/azureblob.(*Object).readMetaData.func1
         0     0% 98.02%     0.33GB  5.83%  github.com/rclone/rclone/backend/azureblob.httpClientFactory.func1.1
         0     0% 98.02%     0.04GB  0.64%  github.com/rclone/rclone/cmd/mount.(*Dir).Lookup
         0     0% 98.02%     2.70GB 47.05%  github.com/rclone/rclone/cmd/mount.(*FileHandle).Flush
         0     0% 98.02%     0.25GB  4.42%  github.com/rclone/rclone/cmd/mount.mount.func1
         0     0% 98.02%     1.34GB 23.37%  github.com/rclone/rclone/fs.pacerInvoker
         0     0% 98.02%     0.33GB  5.83%  github.com/rclone/rclone/fs/fshttp.(*Transport).RoundTrip
         0     0% 98.02%     2.60GB 45.38%  github.com/rclone/rclone/fs/operations.Copy
         0     0% 98.02%     0.90GB 15.67%  github.com/rclone/rclone/lib/pacer.(*Pacer).Call
         0     0% 98.02%     0.44GB  7.65%  github.com/rclone/rclone/lib/pacer.(*Pacer).CallNoRetry (inline)
         0     0% 98.02%     1.34GB 23.37%  github.com/rclone/rclone/lib/pacer.(*Pacer).call
         0     0% 98.02%     0.06GB  1.04%  github.com/rclone/rclone/vendor/bazil.org/fuse.getMessage (inline)
         0     0% 98.02%     0.25GB  4.42%  github.com/rclone/rclone/vendor/bazil.org/fuse/fs.(*Server).Serve
         0     0% 98.02%     4.75GB 82.86%  github.com/rclone/rclone/vendor/bazil.org/fuse/fs.(*Server).Serve.func1
         0     0% 98.02%     4.75GB 82.86%  github.com/rclone/rclone/vendor/bazil.org/fuse/fs.(*Server).handleRequest
         0     0% 98.02%     4.75GB 82.86%  github.com/rclone/rclone/vendor/bazil.org/fuse/fs.(*Server).serve
         0     0% 98.02%     1.02GB 17.77%  github.com/rclone/rclone/vendor/github.com/Azure/azure-pipeline-go/pipeline.(*pipeline).Do
         0     0% 98.02%     1.02GB 17.77%  github.com/rclone/rclone/vendor/github.com/Azure/azure-pipeline-go/pipeline.PolicyFunc.Do
         0     0% 98.02%     0.37GB  6.48%  github.com/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.(*SharedKeyCredential).New.func1
         0     0% 98.02%     0.11GB  1.99%  github.com/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.(*uploadStreamToBlockBlobOptions).end
         0     0% 98.02%     0.85GB 14.90%  github.com/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.BlobURL.GetProperties
         0     0% 98.02%     0.11GB  1.98%  github.com/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.BlockBlobURL.Upload
         0     0% 98.02%     0.04GB  0.76%  github.com/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.ContainerURL.ListBlobsHierarchySegment
         0     0% 98.02%     0.33GB  5.83%  github.com/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.NewRequestLogPolicyFactory.func1.1
         0     0% 98.02%     1.02GB 17.77%  github.com/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.NewRetryPolicyFactory.func1.1
         0     0% 98.02%     1.02GB 17.77%  github.com/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.NewTelemetryPolicyFactory.func1.1
         0     0% 98.02%     1.02GB 17.77%  github.com/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.NewUniqueRequestIDPolicyFactory.func1.1
         0     0% 98.02%     0.45GB  7.82%  github.com/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.UploadStreamToBlockBlob
         0     0% 98.02%     0.85GB 14.90%  github.com/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.blobClient.GetProperties
         0     0% 98.02%     0.11GB  1.98%  github.com/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.blockBlobClient.Upload
         0     0% 98.02%     0.04GB  0.76%  github.com/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.containerClient.ListBlobHierarchySegment
         0     0% 98.02%     0.04GB  0.63%  github.com/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.containerClient.listBlobHierarchySegmentResponder
         0     0% 98.02%     0.37GB  6.46%  github.com/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.responderPolicy.Do
         0     0% 98.02%     0.45GB  7.82%  github.com/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.uploadStream
         0     0% 98.02%     0.68GB 11.85%  github.com/rclone/rclone/vfs.(*Dir).Create
         0     0% 98.02%     0.04GB  0.64%  github.com/rclone/rclone/vfs.(*Dir).Stat
         0     0% 98.02%     0.04GB  0.75%  github.com/rclone/rclone/vfs.(*Dir)._readDir
         0     0% 98.02%     0.04GB  0.64%  github.com/rclone/rclone/vfs.(*Dir).stat
         0     0% 98.02%     1.29GB 22.52%  github.com/rclone/rclone/vfs.(*File).Open
         0     0% 98.02%     0.37GB  6.52%  github.com/rclone/rclone/vfs.(*File).Path
         0     0% 98.02%     1.29GB 22.52%  github.com/rclone/rclone/vfs.(*File).openRW
         0     0% 98.02%     0.52GB  9.04%  github.com/rclone/rclone/vfs.(*File).osPath
         0     0% 98.02%     2.70GB 47.05%  github.com/rclone/rclone/vfs.(*RWFileHandle).Flush
         0     0% 98.02%     2.70GB 47.05%  github.com/rclone/rclone/vfs.(*RWFileHandle).flushWrites
         0     0% 98.02%     1.04GB 18.17%  github.com/rclone/rclone/vfs.(*RWFileHandle).openPending
         0     0% 98.02%     2.32GB 40.53%  github.com/rclone/rclone/vfs/vfscache.(*Cache).Store
         0     0% 98.02%     0.52GB  9.04%  github.com/rclone/rclone/vfs/vfscache.(*Cache).ToOSPath (inline)
         0     0% 98.02%     2.32GB 40.53%  github.com/rclone/rclone/vfs/vfscache.copyObj
         0     0% 98.02%     0.33GB  5.83%  net/http.(*Client).Do (inline)
         0     0% 98.02%     0.33GB  5.83%  net/http.(*Client).do
         0     0% 98.02%     0.33GB  5.83%  net/http.(*Client).send
         0     0% 98.02%     0.33GB  5.83%  net/http.(*Transport).RoundTrip (inline)
         0     0% 98.02%     0.33GB  5.83%  net/http.(*Transport).roundTrip
         0     0% 98.02%     0.34GB  5.95%  net/http.(*persistConn).readLoop
         0     0% 98.02%     0.34GB  5.95%  net/http.(*persistConn).readResponse
         0     0% 98.02%     0.34GB  5.95%  net/http.ReadResponse
         0     0% 98.02%     0.33GB  5.83%  net/http.send
         0     0% 98.02%     0.31GB  5.36%  os.OpenFile
         0     0% 98.02%     0.31GB  5.36%  os.openFileNolog
         0     0% 98.02%     0.38GB  6.55%  path.Join
         0     0% 98.02%     0.52GB  9.04%  path/filepath.Join (inline)
         0     0% 98.02%     0.52GB  9.04%  path/filepath.join
         0     0% 98.02%     0.90GB 15.62%  strings.(*Builder).Grow (inline)
         0     0% 98.02%     0.89GB 15.59%  strings.Join
         0     0% 98.02%     0.06GB  1.04%  sync.(*Pool).Get

Can you try this one?

v1.54.0-beta.5061.705d00b8b.fix-azureblob-memory on branch fix-azureblob-memory (uploaded in 15-30 mins)

This contains one of the potential fixes to the azureblob SDK. That fix isn't merged yet but it looks good to me.

[root@AZ1VNA01 ~]# rclone version
rclone v1.54.0-beta.5061.705d00b8b.fix-azureblob-memory
- os/arch: linux/amd64
- go version: go1.15.6

[root@AZ1VNA01 ~]# go tool pprof -text http://localhost:5572/debug/pprof/heap
Fetching profile over HTTP from http://localhost:5572/debug/pprof/heap
Saved profile in /root/pprof/pprof.rclone.alloc_objects.alloc_space.inuse_objects.inuse_space.042.pb.gz
File: rclone
Type: inuse_space
Time: Jan 15, 2021 at 9:15am (EST)
Showing nodes accounting for 161.81MB, 96.71% of 167.32MB total
Dropped 42 nodes (cum <= 0.84MB)
      flat  flat%   sum%        cum   cum%
   28.47MB 17.01% 17.01%    28.47MB 17.01%  bazil.org/fuse.allocMessage
      23MB 13.75% 30.76%       23MB 13.75%  github.com/rclone/rclone/vfs.newDir
   20.51MB 12.26% 43.02%    20.51MB 12.26%  github.com/Azure/azure-storage-blob-go/azblob.BlobGetPropertiesResponse.NewMetadata
       9MB  5.38% 48.40%       13MB  7.77%  github.com/rclone/rclone/vfs/vfscache.newItem
       9MB  5.38% 53.78%        9MB  5.38%  github.com/rclone/rclone/vfs.newFile
    7.50MB  4.48% 58.26%     7.50MB  4.48%  github.com/rclone/rclone/fs.NewDir (inline)
    6.50MB  3.89% 62.15%     9.50MB  5.68%  context.WithDeadline
    6.22MB  3.72% 65.86%     6.22MB  3.72%  github.com/rclone/rclone/vfs.(*Dir).addObject
       6MB  3.59% 69.45%    41.51MB 24.81%  github.com/rclone/rclone/backend/azureblob.(*Fs).Put
    5.50MB  3.29% 72.74%     5.50MB  3.29%  encoding/xml.copyValue
       5MB  2.99% 75.73%        5MB  2.99%  strings.(*Builder).grow (inline)
    4.57MB  2.73% 78.46%    28.08MB 16.78%  github.com/rclone/rclone/vfs.(*Dir)._readDirFromEntries
    4.50MB  2.69% 81.15%     4.50MB  2.69%  context.(*cancelCtx).Done
       4MB  2.39% 83.54%        4MB  2.39%  sync.NewCond (inline)
       4MB  2.39% 85.93%        4MB  2.39%  net/textproto.(*Reader).ReadMIMEHeader
    3.50MB  2.09% 88.02%    31.97MB 19.11%  bazil.org/fuse.(*Conn).ReadRequest
       3MB  1.79% 89.82%        3MB  1.79%  time.AfterFunc
       3MB  1.79% 91.61%     9.72MB  5.81%  github.com/rclone/rclone/vfs.newRWFileHandle
    2.78MB  1.66% 93.27%     2.78MB  1.66%  bazil.org/fuse/fs.(*Server).saveNode
    2.50MB  1.49% 94.77%    39.48MB 23.59%  github.com/rclone/rclone/cmd/mount.(*Dir).Create
    1.50MB   0.9% 95.66%     1.50MB   0.9%  time.NewTimer
       1MB   0.6% 96.26%        1MB   0.6%  context.WithCancel
    0.75MB  0.45% 96.71%    13.75MB  8.22%  github.com/rclone/rclone/vfs/vfscache.(*Cache)._get
         0     0% 96.71%    28.47MB 17.01%  bazil.org/fuse.getMessage (inline)
         0     0% 96.71%    31.97MB 19.11%  bazil.org/fuse/fs.(*Server).Serve
         0     0% 96.71%    78.84MB 47.12%  bazil.org/fuse/fs.(*Server).Serve.func1
         0     0% 96.71%    78.84MB 47.12%  bazil.org/fuse/fs.(*Server).handleRequest
         0     0% 96.71%     2.78MB  1.66%  bazil.org/fuse/fs.(*Server).saveLookup
         0     0% 96.71%    78.84MB 47.12%  bazil.org/fuse/fs.(*Server).serve
         0     0% 96.71%     9.50MB  5.68%  context.WithTimeout
         0     0% 96.71%        1MB   0.6%  crypto/tls.(*Conn).Handshake
         0     0% 96.71%        1MB   0.6%  crypto/tls.(*Conn).clientHandshake
         0     0% 96.71%        1MB   0.6%  crypto/tls.(*clientHandshakeState).doFullHandshake
         0     0% 96.71%        1MB   0.6%  crypto/tls.(*clientHandshakeState).handshake
         0     0% 96.71%     5.50MB  3.29%  encoding/xml.(*Decoder).Decode (inline)
         0     0% 96.71%     5.50MB  3.29%  encoding/xml.(*Decoder).DecodeElement
         0     0% 96.71%     5.50MB  3.29%  encoding/xml.(*Decoder).unmarshal
         0     0% 96.71%     5.50MB  3.29%  encoding/xml.(*Decoder).unmarshalPath
         0     0% 96.71%     5.50MB  3.29%  encoding/xml.Unmarshal
         0     0% 96.71%       21MB 12.55%  github.com/Azure/azure-pipeline-go/pipeline.(*pipeline).Do
         0     0% 96.71%       21MB 12.55%  github.com/Azure/azure-pipeline-go/pipeline.PolicyFunc.Do
         0     0% 96.71%    11.50MB  6.87%  github.com/Azure/azure-storage-blob-go/azblob.(*SharedKeyCredential).New.func1
         0     0% 96.71%        1MB   0.6%  github.com/Azure/azure-storage-blob-go/azblob.(*copier).close
         0     0% 96.71%        1MB   0.6%  github.com/Azure/azure-storage-blob-go/azblob.(*copier).write
         0     0% 96.71%        1MB   0.6%  github.com/Azure/azure-storage-blob-go/azblob.(*copier).writer
         0     0% 96.71%       13MB  7.77%  github.com/Azure/azure-storage-blob-go/azblob.BlobURL.GetProperties
         0     0% 96.71%        1MB   0.6%  github.com/Azure/azure-storage-blob-go/azblob.BlockBlobURL.CommitBlockList
         0     0% 96.71%        1MB   0.6%  github.com/Azure/azure-storage-blob-go/azblob.BlockBlobURL.StageBlock
         0     0% 96.71%        6MB  3.59%  github.com/Azure/azure-storage-blob-go/azblob.ContainerURL.ListBlobsHierarchySegment
         0     0% 96.71%        6MB  3.59%  github.com/Azure/azure-storage-blob-go/azblob.NewRequestLogPolicyFactory.func1.1
         0     0% 96.71%       21MB 12.55%  github.com/Azure/azure-storage-blob-go/azblob.NewRetryPolicyFactory.func1.1
         0     0% 96.71%       21MB 12.55%  github.com/Azure/azure-storage-blob-go/azblob.NewTelemetryPolicyFactory.func1.1
         0     0% 96.71%       21MB 12.55%  github.com/Azure/azure-storage-blob-go/azblob.NewUniqueRequestIDPolicyFactory.func1.1
         0     0% 96.71%        2MB  1.20%  github.com/Azure/azure-storage-blob-go/azblob.UploadStreamToBlockBlob
         0     0% 96.71%       13MB  7.77%  github.com/Azure/azure-storage-blob-go/azblob.blobClient.GetProperties
         0     0% 96.71%        1MB   0.6%  github.com/Azure/azure-storage-blob-go/azblob.blockBlobClient.CommitBlockList
         0     0% 96.71%        1MB   0.6%  github.com/Azure/azure-storage-blob-go/azblob.blockBlobClient.StageBlock
         0     0% 96.71%        6MB  3.59%  github.com/Azure/azure-storage-blob-go/azblob.containerClient.ListBlobHierarchySegment
         0     0% 96.71%     5.50MB  3.29%  github.com/Azure/azure-storage-blob-go/azblob.containerClient.listBlobHierarchySegmentResponder
         0     0% 96.71%        2MB  1.20%  github.com/Azure/azure-storage-blob-go/azblob.copyFromReader
         0     0% 96.71%    11.50MB  6.87%  github.com/Azure/azure-storage-blob-go/azblob.responderPolicy.Do
         0     0% 96.71%       14MB  8.37%  github.com/rclone/rclone/backend/azureblob.(*Fs).List
         0     0% 96.71%        8MB  4.78%  github.com/rclone/rclone/backend/azureblob.(*Fs).itemToDirEntry
         0     0% 96.71%       14MB  8.37%  github.com/rclone/rclone/backend/azureblob.(*Fs).list
         0     0% 96.71%        6MB  3.59%  github.com/rclone/rclone/backend/azureblob.(*Fs).list.func1
         0     0% 96.71%       14MB  8.37%  github.com/rclone/rclone/backend/azureblob.(*Fs).listDir
         0     0% 96.71%        8MB  4.78%  github.com/rclone/rclone/backend/azureblob.(*Fs).listDir.func1
         0     0% 96.71%    36.01MB 21.52%  github.com/rclone/rclone/backend/azureblob.(*Object).Update
         0     0% 96.71%        2MB  1.20%  github.com/rclone/rclone/backend/azureblob.(*Object).Update.func1
         0     0% 96.71%    21.01MB 12.55%  github.com/rclone/rclone/backend/azureblob.(*Object).decodeMetaDataFromPropertiesResponse
         0     0% 96.71%    34.01MB 20.32%  github.com/rclone/rclone/backend/azureblob.(*Object).readMetaData
         0     0% 96.71%       13MB  7.77%  github.com/rclone/rclone/backend/azureblob.(*Object).readMetaData.func1
         0     0% 96.71%        6MB  3.59%  github.com/rclone/rclone/backend/azureblob.httpClientFactory.func1.1
         0     0% 96.71%    35.58MB 21.26%  github.com/rclone/rclone/cmd/mount.(*Dir).Lookup
         0     0% 96.71%        1MB   0.6%  github.com/rclone/rclone/cmd/mount.(*Dir).ReadDirAll
         0     0% 96.71%    31.97MB 19.11%  github.com/rclone/rclone/cmd/mount.mount.func2
         0     0% 96.71%       21MB 12.55%  github.com/rclone/rclone/fs.pacerInvoker
         0     0% 96.71%        6MB  3.59%  github.com/rclone/rclone/fs/fshttp.(*Transport).RoundTrip
         0     0% 96.71%     8.50MB  5.08%  github.com/rclone/rclone/fs/list.DirSorted
         0     0% 96.71%    41.51MB 24.81%  github.com/rclone/rclone/fs/operations.Copy
         0     0% 96.71%       19MB 11.36%  github.com/rclone/rclone/lib/pacer.(*Pacer).Call
         0     0% 96.71%        2MB  1.20%  github.com/rclone/rclone/lib/pacer.(*Pacer).CallNoRetry (inline)
         0     0% 96.71%       21MB 12.55%  github.com/rclone/rclone/lib/pacer.(*Pacer).call
         0     0% 96.71%     8.50MB  5.08%  github.com/rclone/rclone/vfs.(*Dir).Create
         0     0% 96.71%        1MB   0.6%  github.com/rclone/rclone/vfs.(*Dir).ReadDirAll
         0     0% 96.71%    35.58MB 21.26%  github.com/rclone/rclone/vfs.(*Dir).Stat
         0     0% 96.71%    36.58MB 21.86%  github.com/rclone/rclone/vfs.(*Dir)._readDir
         0     0% 96.71%    35.58MB 21.26%  github.com/rclone/rclone/vfs.(*Dir).stat
         0     0% 96.71%    28.48MB 17.02%  github.com/rclone/rclone/vfs.(*File).Open
         0     0% 96.71%        5MB  2.99%  github.com/rclone/rclone/vfs.(*File).Path
         0     0% 96.71%     9.72MB  5.81%  github.com/rclone/rclone/vfs.(*File).openRW
         0     0% 96.71%     6.22MB  3.72%  github.com/rclone/rclone/vfs.(*RWFileHandle).Truncate
         0     0% 96.71%     6.22MB  3.72%  github.com/rclone/rclone/vfs.(*RWFileHandle).openPending
         0     0% 96.71%    13.75MB  8.22%  github.com/rclone/rclone/vfs/vfscache.(*Cache).Exists
         0     0% 96.71%    13.75MB  8.22%  github.com/rclone/rclone/vfs/vfscache.(*Cache).get
         0     0% 96.71%    37.51MB 22.42%  github.com/rclone/rclone/vfs/vfscache.(*Item).Close.func2
         0     0% 96.71%    37.51MB 22.42%  github.com/rclone/rclone/vfs/vfscache.(*Item)._store
         0     0% 96.71%    37.51MB 22.42%  github.com/rclone/rclone/vfs/vfscache.(*Item).store
         0     0% 96.71%    37.51MB 22.42%  github.com/rclone/rclone/vfs/vfscache/writeback.(*WriteBack).upload
         0     0% 96.71%        6MB  3.59%  net/http.(*Client).Do (inline)
         0     0% 96.71%        6MB  3.59%  net/http.(*Client).do
         0     0% 96.71%        6MB  3.59%  net/http.(*Client).send
         0     0% 96.71%        6MB  3.59%  net/http.(*Transport).RoundTrip (inline)
         0     0% 96.71%        6MB  3.59%  net/http.(*Transport).roundTrip
         0     0% 96.71%        1MB   0.6%  net/http.(*persistConn).addTLS.func2
         0     0% 96.71%        4MB  2.39%  net/http.(*persistConn).readLoop
         0     0% 96.71%        4MB  2.39%  net/http.(*persistConn).readResponse
         0     0% 96.71%     1.50MB   0.9%  net/http.(*persistConn).roundTrip
         0     0% 96.71%        4MB  2.39%  net/http.ReadResponse
         0     0% 96.71%        6MB  3.59%  net/http.send
         0     0% 96.71%        5MB  2.99%  path.Join
         0     0% 96.71%     1.50MB   0.9%  runtime.doInit
         0     0% 96.71%     1.50MB   0.9%  runtime.main
         0     0% 96.71%        5MB  2.99%  strings.(*Builder).Grow (inline)
         0     0% 96.71%        5MB  2.99%  strings.Join
         0     0% 96.71%    28.47MB 17.01%  sync.(*Pool).Get

Uptime 2h

HTOP:
3266535 root 20 0 1081M 418M 31500 S 6.6 1.3 16:32.95 rclone mount --log-file=/var/log/dcmsys/rclone.log --log-level ERROR --config=/etc/rclone.conf --allow-other --default-permissions --max-read-ahead=4M --vfs-cache-mode=writes --vfs-cache-max-size=1G --azureblob-memory-pool-f

That looks much better - 20MB in use by the azblob sdk.

I need this to be merged upstream before I can include it in rclone, but hopefully that will be in time for 1.54

uptime 3 days
rclone version
rclone v1.54.0-beta.5061.705d00b8b.fix-azureblob-memory
- os/arch: linux/amd64
- go version: go1.15.6

HTOP:
3266535 root 20 0 4105M 3255M 10164 S 0.0 10.1 11h02:02 rclone mount --log-file=/var/log/dcmsys/rclone.log --log-level ERROR --config=/etc/rclone.conf --allow-other --default-permissions --max-read-ahead=4M --vfs-cache-mode=writes --vfs-ca

Fetching profile over HTTP from http://localhost:5572/debug/pprof/heap
Saved profile in /root/pprof/pprof.rclone.alloc_objects.alloc_space.inuse_objects.inuse_space.044.pb.gz
File: rclone
Type: inuse_space
Time: Jan 18, 2021 at 10:20am (EST)
Showing nodes accounting for 2586.17MB, 98.67% of 2620.96MB total
Dropped 52 nodes (cum <= 13.10MB)
      flat  flat%   sum%        cum   cum%
  451.11MB 17.21% 17.21%   451.61MB 17.23%  github.com/Azure/azure-storage-blob-go/azblob.BlobGetPropertiesResponse.NewMetadata
  325.57MB 12.42% 29.63%   406.57MB 15.51%  github.com/rclone/rclone/vfs/vfscache.newItem
  252.55MB  9.64% 39.27%   252.55MB  9.64%  github.com/rclone/rclone/vfs.newFile
  187.52MB  7.15% 46.42%   187.52MB  7.15%  context.(*cancelCtx).Done
  186.52MB  7.12% 53.54%   333.13MB 12.71%  context.WithDeadline
  184.52MB  7.04% 60.58%   184.52MB  7.04%  strings.(*Builder).grow (inline)
  159.31MB  6.08% 66.66%   159.31MB  6.08%  github.com/rclone/rclone/vfs.(*Dir).addObject
  155.01MB  5.91% 72.57%   155.01MB  5.91%  net/textproto.(*Reader).ReadMIMEHeader
  144.52MB  5.51% 78.09%  1063.71MB 40.58%  github.com/rclone/rclone/backend/azureblob.(*Fs).Put
  124.01MB  4.73% 82.82%   131.11MB  5.00%  time.AfterFunc
   88.51MB  3.38% 86.19%    88.51MB  3.38%  bazil.org/fuse.(*Conn).ReadRequest
      81MB  3.09% 89.29%       81MB  3.09%  sync.NewCond (inline)
      78MB  2.98% 92.26%   255.81MB  9.76%  github.com/rclone/rclone/vfs.newRWFileHandle
      41MB  1.56% 93.83%       41MB  1.56%  encoding/base64.(*Encoding).EncodeToString
   36.51MB  1.39% 95.22%    36.51MB  1.39%  github.com/rclone/rclone/vfs.newDir
   25.50MB  0.97% 96.19%  1112.70MB 42.45%  github.com/rclone/rclone/cmd/mount.(*Dir).Create
   22.50MB  0.86% 97.05%    35.50MB  1.35%  context.WithCancel
   18.50MB  0.71% 97.76%    18.50MB  0.71%  github.com/rclone/rclone/vfs.(*File).addWriter
   15.09MB  0.58% 98.33%    15.09MB  0.58%  time.startTimer
    8.19MB  0.31% 98.64%    43.69MB  1.67%  github.com/rclone/rclone/vfs.(*Dir)._readDirFromEntries
    0.75MB 0.029% 98.67%   407.32MB 15.54%  github.com/rclone/rclone/vfs/vfscache.(*Cache)._get
         0     0% 98.67%    88.51MB  3.38%  bazil.org/fuse/fs.(*Server).Serve
         0     0% 98.67%  1192.68MB 45.51%  bazil.org/fuse/fs.(*Server).Serve.func1
         0     0% 98.67%  1192.68MB 45.51%  bazil.org/fuse/fs.(*Server).handleRequest
         0     0% 98.67%  1192.68MB 45.51%  bazil.org/fuse/fs.(*Server).serve
         0     0% 98.67%   333.13MB 12.71%  context.WithTimeout
         0     0% 98.67%    28.50MB  1.09%  context.propagateCancel
         0     0% 98.67%       15MB  0.57%  encoding/xml.(*Decoder).Decode (inline)
         0     0% 98.67%       15MB  0.57%  encoding/xml.(*Decoder).DecodeElement
         0     0% 98.67%       15MB  0.57%  encoding/xml.(*Decoder).unmarshal
         0     0% 98.67%       15MB  0.57%  encoding/xml.(*Decoder).unmarshalPath
         0     0% 98.67%       15MB  0.57%  encoding/xml.Unmarshal
         0     0% 98.67%   511.58MB 19.52%  github.com/Azure/azure-pipeline-go/pipeline.(*pipeline).Do
         0     0% 98.67%   511.58MB 19.52%  github.com/Azure/azure-pipeline-go/pipeline.PolicyFunc.Do
         0     0% 98.67%   178.45MB  6.81%  github.com/Azure/azure-storage-blob-go/azblob.(*SharedKeyCredential).New.func1
         0     0% 98.67%    31.55MB  1.20%  github.com/Azure/azure-storage-blob-go/azblob.(*copier).close
         0     0% 98.67%    59.49MB  2.27%  github.com/Azure/azure-storage-blob-go/azblob.(*copier).write
         0     0% 98.67%    59.49MB  2.27%  github.com/Azure/azure-storage-blob-go/azblob.(*copier).writer
         0     0% 98.67%   397.53MB 15.17%  github.com/Azure/azure-storage-blob-go/azblob.BlobURL.GetProperties
         0     0% 98.67%    31.55MB  1.20%  github.com/Azure/azure-storage-blob-go/azblob.BlockBlobURL.CommitBlockList
         0     0% 98.67%    59.49MB  2.27%  github.com/Azure/azure-storage-blob-go/azblob.BlockBlobURL.StageBlock
         0     0% 98.67%       23MB  0.88%  github.com/Azure/azure-storage-blob-go/azblob.ContainerURL.ListBlobsHierarchySegment
         0     0% 98.67%   163.45MB  6.24%  github.com/Azure/azure-storage-blob-go/azblob.NewRequestLogPolicyFactory.func1.1
         0     0% 98.67%   511.58MB 19.52%  github.com/Azure/azure-storage-blob-go/azblob.NewRetryPolicyFactory.func1.1
         0     0% 98.67%   511.58MB 19.52%  github.com/Azure/azure-storage-blob-go/azblob.NewTelemetryPolicyFactory.func1.1
         0     0% 98.67%   511.58MB 19.52%  github.com/Azure/azure-storage-blob-go/azblob.NewUniqueRequestIDPolicyFactory.func1.1
         0     0% 98.67%    54.55MB  2.08%  github.com/Azure/azure-storage-blob-go/azblob.UploadStreamToBlockBlob
         0     0% 98.67%   397.53MB 15.17%  github.com/Azure/azure-storage-blob-go/azblob.blobClient.GetProperties
         0     0% 98.67%    31.55MB  1.20%  github.com/Azure/azure-storage-blob-go/azblob.blockBlobClient.CommitBlockList
         0     0% 98.67%    59.49MB  2.27%  github.com/Azure/azure-storage-blob-go/azblob.blockBlobClient.StageBlock
         0     0% 98.67%       23MB  0.88%  github.com/Azure/azure-storage-blob-go/azblob.containerClient.ListBlobHierarchySegment
         0     0% 98.67%       15MB  0.57%  github.com/Azure/azure-storage-blob-go/azblob.containerClient.listBlobHierarchySegmentResponder
         0     0% 98.67%    54.55MB  2.08%  github.com/Azure/azure-storage-blob-go/azblob.copyFromReader
         0     0% 98.67%   178.45MB  6.81%  github.com/Azure/azure-storage-blob-go/azblob.responderPolicy.Do
         0     0% 98.67%    30.50MB  1.16%  github.com/rclone/rclone/backend/azureblob.(*Fs).List
         0     0% 98.67%       35MB  1.34%  github.com/rclone/rclone/backend/azureblob.(*Fs).list
         0     0% 98.67%       23MB  0.88%  github.com/rclone/rclone/backend/azureblob.(*Fs).list.func1
         0     0% 98.67%    31.50MB  1.20%  github.com/rclone/rclone/backend/azureblob.(*Fs).listDir
         0     0% 98.67%   944.70MB 36.04%  github.com/rclone/rclone/backend/azureblob.(*Object).Update
         0     0% 98.67%    54.55MB  2.08%  github.com/rclone/rclone/backend/azureblob.(*Object).Update.func1
         0     0% 98.67%   492.61MB 18.80%  github.com/rclone/rclone/backend/azureblob.(*Object).decodeMetaDataFromPropertiesResponse
         0     0% 98.67%   890.14MB 33.96%  github.com/rclone/rclone/backend/azureblob.(*Object).readMetaData
         0     0% 98.67%   397.53MB 15.17%  github.com/rclone/rclone/backend/azureblob.(*Object).readMetaData.func1
         0     0% 98.67%   163.45MB  6.24%  github.com/rclone/rclone/backend/azureblob.httpClientFactory.func1.1
         0     0% 98.67%    56.13MB  2.14%  github.com/rclone/rclone/cmd/mount.(*Dir).Lookup
         0     0% 98.67%    88.51MB  3.38%  github.com/rclone/rclone/cmd/mount.mount.func2
         0     0% 98.67%   474.59MB 18.11%  github.com/rclone/rclone/fs.pacerInvoker
         0     0% 98.67%   163.45MB  6.24%  github.com/rclone/rclone/fs/fshttp.(*Transport).RoundTrip
         0     0% 98.67%    20.50MB  0.78%  github.com/rclone/rclone/fs/list.DirSorted
         0     0% 98.67%  1080.21MB 41.21%  github.com/rclone/rclone/fs/operations.Copy
         0     0% 98.67%   420.04MB 16.03%  github.com/rclone/rclone/lib/pacer.(*Pacer).Call
         0     0% 98.67%    54.55MB  2.08%  github.com/rclone/rclone/lib/pacer.(*Pacer).CallNoRetry (inline)
         0     0% 98.67%   474.59MB 18.11%  github.com/rclone/rclone/lib/pacer.(*Pacer).call
         0     0% 98.67%   250.55MB  9.56%  github.com/rclone/rclone/vfs.(*Dir).Create
         0     0% 98.67%    56.13MB  2.14%  github.com/rclone/rclone/vfs.(*Dir).Stat
         0     0% 98.67%    61.69MB  2.35%  github.com/rclone/rclone/vfs.(*Dir)._readDir
         0     0% 98.67%    56.13MB  2.14%  github.com/rclone/rclone/vfs.(*Dir).stat
         0     0% 98.67%   845.66MB 32.27%  github.com/rclone/rclone/vfs.(*File).Open
         0     0% 98.67%   182.52MB  6.96%  github.com/rclone/rclone/vfs.(*File).Path
         0     0% 98.67%   255.81MB  9.76%  github.com/rclone/rclone/vfs.(*File).openRW
         0     0% 98.67%   159.31MB  6.08%  github.com/rclone/rclone/vfs.(*RWFileHandle).Truncate
         0     0% 98.67%   159.31MB  6.08%  github.com/rclone/rclone/vfs.(*RWFileHandle).openPending
         0     0% 98.67%   407.32MB 15.54%  github.com/rclone/rclone/vfs/vfscache.(*Cache).Exists
         0     0% 98.67%   407.32MB 15.54%  github.com/rclone/rclone/vfs/vfscache.(*Cache).get
         0     0% 98.67%   945.20MB 36.06%  github.com/rclone/rclone/vfs/vfscache.(*Item).Close.func2
         0     0% 98.67%   949.70MB 36.23%  github.com/rclone/rclone/vfs/vfscache.(*Item)._store
         0     0% 98.67%   945.20MB 36.06%  github.com/rclone/rclone/vfs/vfscache.(*Item).store
         0     0% 98.67%   945.20MB 36.06%  github.com/rclone/rclone/vfs/vfscache/writeback.(*WriteBack).upload
         0     0% 98.67%   163.45MB  6.24%  net/http.(*Client).Do (inline)
         0     0% 98.67%   163.45MB  6.24%  net/http.(*Client).do
         0     0% 98.67%   163.45MB  6.24%  net/http.(*Client).send
         0     0% 98.67%   163.45MB  6.24%  net/http.(*Transport).RoundTrip (inline)
         0     0% 98.67%   163.45MB  6.24%  net/http.(*Transport).roundTrip
         0     0% 98.67%   155.01MB  5.91%  net/http.(*persistConn).readLoop
         0     0% 98.67%   155.01MB  5.91%  net/http.(*persistConn).readResponse
         0     0% 98.67%   155.01MB  5.91%  net/http.ReadResponse
         0     0% 98.67%   163.45MB  6.24%  net/http.send
         0     0% 98.67%   184.02MB  7.02%  path.Join
         0     0% 98.67%   184.52MB  7.04%  strings.(*Builder).Grow (inline)
         0     0% 98.67%   184.02MB  7.02%  strings.Join

Is that acceptable?

Your profile looks like you have lots of objects stored in the VFS cache - that is where the memory is being used I think.

4Gb of ram is OK for me. I need to be sure that the memory consumption won't grow constantly without limits. If consuming won't exceed 8Gb for example, then it's ok.

It is looking OK now I think :crossed_fingers:

Hi Nick,

Not really.
After 9 days of working I see that rclone uses 13Gb of RAM. Could you please recommend me what can I do in order to stop memory consumption going up? I need to have stable RAM consumption. 8Gb is OK, but not more.

uptime: 1 week 2 days

htop:

3266535 root       20   0 12.9G 12.3G 11472 S 362. 39.0 67h34:41 rclone mount --log-file=/var/log/dcmsys/rclone.log --log-level ERROR --config=/etc/rclone.conf --allow-other --default-permissions --max-read-ahead=4M --vfs-cache-mode=writes -
rclone mount --log-file=/var/log/dcmsys/rclone.log --log-level ERROR --config=/etc/rclone.conf --allow-other --default-permissions --max-read-ahead=4M --vfs-cache-mode=writes --vfs-cache-max-size=1G --azureblob-memory-pool-flush-time=1m0s --buffer-size=0 --azureblob-memory-pool-use-mmap --use-mmap --dir-cache-time=2h --timeout=1h --cache-dir=/var/dcmsys/cache --uid=1100 --gid=1100 --cache-chunk-no-memory --rc --size-only --allow-non-empty blob:studiesh /var/dcmsys/storages/GCP
go tool pprof -text http://localhost:5572/debug/pprof/heap
Fetching profile over HTTP from http://localhost:5572/debug/pprof/heap
Saved profile in /root/pprof/pprof.rclone.alloc_objects.alloc_space.inuse_objects.inuse_space.046.pb.gz
File: rclone
Type: inuse_space
Time: Jan 24, 2021 at 2:55pm (EST)
Showing nodes accounting for 10948.52MB, 98.53% of 11111.98MB total
Dropped 200 nodes (cum <= 55.56MB)
      flat  flat%   sum%        cum   cum%
 1983.98MB 17.85% 17.85%  1984.48MB 17.86%  github.com/Azure/azure-storage-blob-go/azblob.BlobGetPropertiesResponse.NewMetadata
 1379.29MB 12.41% 30.27%  1762.82MB 15.86%  github.com/rclone/rclone/vfs/vfscache.newItem
 1133.71MB 10.20% 40.47%  1133.71MB 10.20%  github.com/rclone/rclone/vfs.newFile
  821.60MB  7.39% 47.86%   821.60MB  7.39%  strings.(*Builder).grow (inline)
  717.56MB  6.46% 54.32%  1271.10MB 11.44%  context.WithDeadline
  690.06MB  6.21% 60.53%   690.06MB  6.21%  context.(*cancelCtx).Done
  678.52MB  6.11% 66.64%   678.52MB  6.11%  net/textproto.(*Reader).ReadMIMEHeader
  674.57MB  6.07% 72.71%  4686.06MB 42.17%  github.com/rclone/rclone/backend/azureblob.(*Fs).Put
  663.85MB  5.97% 78.68%   663.85MB  5.97%  github.com/rclone/rclone/vfs.(*Dir).addObject
  525.54MB  4.73% 83.41%   539.09MB  4.85%  time.AfterFunc
  416.53MB  3.75% 87.16%   427.84MB  3.85%  bazil.org/fuse.(*Conn).ReadRequest
  407.52MB  3.67% 90.83%  1156.83MB 10.41%  github.com/rclone/rclone/vfs.newRWFileHandle
  382.52MB  3.44% 94.27%   382.52MB  3.44%  sync.NewCond (inline)
  194.51MB  1.75% 96.02%   194.51MB  1.75%  encoding/base64.(*Encoding).EncodeToString
   93.50MB  0.84% 96.86%  4940.71MB 44.46%  github.com/rclone/rclone/cmd/mount.(*Dir).Create
      86MB  0.77% 97.64%       86MB  0.77%  github.com/rclone/rclone/vfs.(*File).addWriter
      61MB  0.55% 98.18%       61MB  0.55%  time.startTimer
      33MB   0.3% 98.48%    61.50MB  0.55%  context.WithCancel
    1.25MB 0.011% 98.49%  1764.07MB 15.88%  github.com/rclone/rclone/vfs/vfscache.(*Cache)._get
       1MB 0.009% 98.50%   702.46MB  6.32%  github.com/Azure/azure-storage-blob-go/azblob.(*SharedKeyCredential).New.func1
       1MB 0.009% 98.51%   119.52MB  1.08%  github.com/Azure/azure-storage-blob-go/azblob.copyFromReader
       1MB 0.009% 98.52%  5066.74MB 45.60%  bazil.org/fuse/fs.(*Server).handleRequest
       1MB 0.009% 98.53%  1769.09MB 15.92%  github.com/Azure/azure-storage-blob-go/azblob.blobClient.GetProperties
         0     0% 98.53%   427.84MB  3.85%  bazil.org/fuse/fs.(*Server).Serve
         0     0% 98.53%  5067.24MB 45.60%  bazil.org/fuse/fs.(*Server).Serve.func1
         0     0% 98.53%  5067.24MB 45.60%  bazil.org/fuse/fs.(*Server).serve
         0     0% 98.53%  1271.10MB 11.44%  context.WithTimeout
         0     0% 98.53%    57.01MB  0.51%  context.propagateCancel
         0     0% 98.53%  1978.06MB 17.80%  github.com/Azure/azure-pipeline-go/pipeline.(*pipeline).Do
         0     0% 98.53%  1976.56MB 17.79%  github.com/Azure/azure-pipeline-go/pipeline.PolicyFunc.Do
         0     0% 98.53%    70.51MB  0.63%  github.com/Azure/azure-storage-blob-go/azblob.(*copier).close
         0     0% 98.53%    98.45MB  0.89%  github.com/Azure/azure-storage-blob-go/azblob.(*copier).write
         0     0% 98.53%    98.45MB  0.89%  github.com/Azure/azure-storage-blob-go/azblob.(*copier).writer
         0     0% 98.53%  1769.09MB 15.92%  github.com/Azure/azure-storage-blob-go/azblob.BlobURL.GetProperties
         0     0% 98.53%    70.51MB  0.63%  github.com/Azure/azure-storage-blob-go/azblob.BlockBlobURL.CommitBlockList
         0     0% 98.53%    98.45MB  0.89%  github.com/Azure/azure-storage-blob-go/azblob.BlockBlobURL.StageBlock
         0     0% 98.53%   674.45MB  6.07%  github.com/Azure/azure-storage-blob-go/azblob.NewRequestLogPolicyFactory.func1.1
         0     0% 98.53%  1976.06MB 17.78%  github.com/Azure/azure-storage-blob-go/azblob.NewRetryPolicyFactory.func1.1
         0     0% 98.53%  1976.56MB 17.79%  github.com/Azure/azure-storage-blob-go/azblob.NewTelemetryPolicyFactory.func1.1
         0     0% 98.53%  1976.56MB 17.79%  github.com/Azure/azure-storage-blob-go/azblob.NewUniqueRequestIDPolicyFactory.func1.1
         0     0% 98.53%   119.52MB  1.08%  github.com/Azure/azure-storage-blob-go/azblob.UploadStreamToBlockBlob
         0     0% 98.53%    70.51MB  0.63%  github.com/Azure/azure-storage-blob-go/azblob.blockBlobClient.CommitBlockList
         0     0% 98.53%    98.45MB  0.89%  github.com/Azure/azure-storage-blob-go/azblob.blockBlobClient.StageBlock
         0     0% 98.53%   698.45MB  6.29%  github.com/Azure/azure-storage-blob-go/azblob.responderPolicy.Do
         0     0% 98.53%    60.51MB  0.54%  github.com/rclone/rclone/backend/azureblob.(*Fs).list
         0     0% 98.53%  4062.49MB 36.56%  github.com/rclone/rclone/backend/azureblob.(*Object).Update
         0     0% 98.53%   119.52MB  1.08%  github.com/rclone/rclone/backend/azureblob.(*Object).Update.func1
         0     0% 98.53%  2178.48MB 19.60%  github.com/rclone/rclone/backend/azureblob.(*Object).decodeMetaDataFromPropertiesResponse
         0     0% 98.53%  3930.63MB 35.37%  github.com/rclone/rclone/backend/azureblob.(*Object).readMetaData
         0     0% 98.53%  1769.09MB 15.92%  github.com/rclone/rclone/backend/azureblob.(*Object).readMetaData.func1
         0     0% 98.53%   674.45MB  6.07%  github.com/rclone/rclone/backend/azureblob.httpClientFactory.func1.1
         0     0% 98.53%    69.14MB  0.62%  github.com/rclone/rclone/cmd/mount.(*Dir).Lookup
         0     0% 98.53%   427.84MB  3.85%  github.com/rclone/rclone/cmd/mount.mount.func2
         0     0% 98.53%  1933.11MB 17.40%  github.com/rclone/rclone/fs.pacerInvoker
         0     0% 98.53%   672.95MB  6.06%  github.com/rclone/rclone/fs/fshttp.(*Transport).RoundTrip
         0     0% 98.53%  4717.06MB 42.45%  github.com/rclone/rclone/fs/operations.Copy
         0     0% 98.53%  1813.10MB 16.32%  github.com/rclone/rclone/lib/pacer.(*Pacer).Call
         0     0% 98.53%   118.52MB  1.07%  github.com/rclone/rclone/lib/pacer.(*Pacer).CallNoRetry (inline)
         0     0% 98.53%  1933.11MB 17.40%  github.com/rclone/rclone/lib/pacer.(*Pacer).call
         0     0% 98.53%  1131.21MB 10.18%  github.com/rclone/rclone/vfs.(*Dir).Create
         0     0% 98.53%    69.14MB  0.62%  github.com/rclone/rclone/vfs.(*Dir).Stat
         0     0% 98.53%    77.70MB   0.7%  github.com/rclone/rclone/vfs.(*Dir)._readDir
         0     0% 98.53%    69.14MB  0.62%  github.com/rclone/rclone/vfs.(*Dir).stat
         0     0% 98.53%  3732.50MB 33.59%  github.com/rclone/rclone/vfs.(*File).Open
         0     0% 98.53%   811.60MB  7.30%  github.com/rclone/rclone/vfs.(*File).Path
         0     0% 98.53%  1156.83MB 10.41%  github.com/rclone/rclone/vfs.(*File).openRW
         0     0% 98.53%   663.33MB  5.97%  github.com/rclone/rclone/vfs.(*RWFileHandle).Truncate
         0     0% 98.53%   663.33MB  5.97%  github.com/rclone/rclone/vfs.(*RWFileHandle).openPending
         0     0% 98.53%  1764.07MB 15.88%  github.com/rclone/rclone/vfs/vfscache.(*Cache).Exists
         0     0% 98.53%  1764.07MB 15.88%  github.com/rclone/rclone/vfs/vfscache.(*Cache).get
         0     0% 98.53%  4125.51MB 37.13%  github.com/rclone/rclone/vfs/vfscache.(*Item).Close.func2
         0     0% 98.53%  4132.51MB 37.19%  github.com/rclone/rclone/vfs/vfscache.(*Item)._store
         0     0% 98.53%  4125.51MB 37.13%  github.com/rclone/rclone/vfs/vfscache.(*Item).store
         0     0% 98.53%  4125.51MB 37.13%  github.com/rclone/rclone/vfs/vfscache/writeback.(*WriteBack).upload
         0     0% 98.53%   674.45MB  6.07%  net/http.(*Client).Do (inline)
         0     0% 98.53%   674.45MB  6.07%  net/http.(*Client).do
         0     0% 98.53%   672.95MB  6.06%  net/http.(*Client).send
         0     0% 98.53%   672.95MB  6.06%  net/http.(*Transport).RoundTrip (inline)
         0     0% 98.53%   672.95MB  6.06%  net/http.(*Transport).roundTrip
         0     0% 98.53%   679.02MB  6.11%  net/http.(*persistConn).readLoop
         0     0% 98.53%   679.02MB  6.11%  net/http.(*persistConn).readResponse
         0     0% 98.53%   679.02MB  6.11%  net/http.ReadResponse
         0     0% 98.53%   672.95MB  6.06%  net/http.send
         0     0% 98.53%   818.60MB  7.37%  path.Join
         0     0% 98.53%   821.60MB  7.39%  strings.(*Builder).Grow (inline)
         0     0% 98.53%   821.10MB  7.39%  strings.Join

Best,
Raymond

What you can see in this memory profile is that the memory is used by the VFS cache.

So it looks like the azure blob SDK leak is fixed, but you have a lot of cached items in the VFS cache.

I think this is the same issue as

How many items do you have in the VFS cache?