Ask for settings recommendation for azureblob

Raimondas · January 6, 2021, 7:31pm

Hi,
I'm trying to implement a high load system with cloud based storage.

What is the problem you are having with rclone?

currently I have a VM system with 32Gb ram on board and 4 CPU cores. RAM and CPU consumption of rclone is way to high. rclone consumed 18Gb of RAM after 6h of working.
CPU is jumping from 20 to 100%.
Could you please give me an advice what settings should I use for mounting azure blob in order to keep CPU and RAM for rclone on mininum, or at least - 10Gb max RAM.
I can pay some money for help (tuning the system/config) if you exactly know what to do, or you are the rclone developer.

What is your rclone version (output from `rclone version`)

rclone v1.53.3

Which OS you are using and how many bits (eg Windows 7, 64 bit)

CentOS 8.2.2004 (4.18.0-193.el8.x86_64)

Which cloud storage system are you using? (eg Google Drive)

Azure blob storage

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

I am using following line to mount disk:

rclone mount --log-file=/var/log/dcmsys/rclone.log --log-level ERROR --config=/etc/rclone.conf --allow-other --default-permissions --max-read-ahead=16M --vfs-cache-mode=writes --vfs-cache-max-size=10G --azureblob-memory-pool-use-mmap --use-mmap --cache-chunk-no-memory --cache-dir=/var/dcmsys/cache --uid=1100 --gid=1100 --allow-non-empty blob:studiesh /var/dcmsys/storages/GCP

next time I'll try following config for fstab:

rclonefs#blob:studiesh  /var/dcmsys/storages/GCP  fuse  _netdev,config=/etc/rclone.conf,allow-other,default-permissions,rw,max-read-ahead=16M,vfs-cache-mode=writes,vfs-cache-max-size=2G,azureblob-memory-pool-flush-time=1m0s,buffer-size=0,azureblob-memory-pool-use-mmap,use-mmap,cache-chunk-no-memory,cache-db-purge,buffer-size=32M,dir-cache-time=72h,timeout=1h,cache-dir=/var/dcmsys/cache,uid=dcmsys,gid=dcmsys,size-only,allow-non-empty   0 0

The rclone config contents with secrets removed.

[blob]
type = azureblob
account = az1vna01sa
key =

asdffdsa · January 6, 2021, 8:10pm

hello and welcome to the forum,

--cache-chunk-no-memory, can be removed, does nothing based on your config and command.
--cache-db-purge, can be removed, does nothing based on your config and command.
--allow-non-empty, enables over-mounting, should be removed unless 1000% sure it is needed.

calisro · January 6, 2021, 8:16pm

How are you measuring memory consumption? It could just be the OS file cache which will just get purged as required.

If you want to reduce CPU, you could reduce the threads with --checkers --transfers. Default is four. (actually you're using a mount so those are ineffective on a mount). You can also use tps_limit to slow rclone down. All of these will have an impact of how fast things will be though. --buffer-size will allocate a maximum per thread and turning that down will reduce memory.

Also noticed you have this twice with two values. Remove one as the last one will be the one its using:

buffer-size=0
buffer-size=32M

What is your use-case for the mount?

Raimondas · January 6, 2021, 8:33pm

My use-case - receive dicom files and upload them to the cloud blob for the long-term storage.
Server should have access to the files as if they are locally located. All the other files, such as DB and index files stored locally on fast ssd disk. In general - dicom files once written will be mostly read and rarely written after that procedure of ingestion. Local VFS cache is for better performance of the server. It should help faster ingest studies and start moving them to cloud. If local cache is full, then the whole ingestion will be paused or slow down the speed of ingestion. It's OK.

Thanks for comments. Next time I'll try following configuration:

rclonefs#blob:studiesh  /var/dcmsys/storages/GCP  fuse  _netdev,config=/etc/rclone.conf,allow-other,default-permissions,rw,max-read-ahead=16M,vfs-cache-mode=writes,vfs-cache-max-size=2G,azureblob-memory-pool-flush-time=1m0s,buffer-size=0,azureblob-memory-pool-use-mmap,use-mmap,dir-cache-time=72h,timeout=1h,cache-dir=/var/dcmsys/cache,uid=dcmsys,gid=dcmsys,size-only,allow-non-empty   0 0

calisro · January 6, 2021, 9:52pm

You may want to consider not writing to your mount. Instead use it as a source for your files only. When uploading them, you'll get far better performance using another rclone move/copy/sync command to move them to the cloud as needed. If that fits your usecase.

If you do that, then you can better manage your transfers/checkers on that second process used to upload the data and make it less impactful to your read-only mount.

ncw · January 7, 2021, 11:36am

It may also be worth trying the latest beta which fixes some memory usage problems with azureblob (where were caused by the Azure SDK).

Raimondas · January 7, 2021, 7:28pm

Hi Nick,

I used following line to mount:

rclone mount --log-file=/var/log/dcmsys/rclone.log --log-level ERROR --config=/etc/rclone.conf --allow-other --default-permissions --max-read-ahead=16M --vfs-cache-mode=writes --vfs-cache-max-size=2G --azureblob-memory-pool-flush-time=1m0s --buffer-size=0 --azureblob-memory-pool-use-mmap --use-mmap --dir-cache-time=72h --timeout=1h --cache-dir=/var/dcmsys/cache --uid=1100 --gid=1100 --size-only --allow-non-empty blob:studiesh /var/dcmsys/storages/GCP

after 5 hours of working, rclone has eaten 12Gb RAM.

I used rclone version:
rclone v1.54.0-beta.5040.71edc75ca

I don't like the result of the modifications. I think I have to disable local caching for rclone if I want to keep ram consumption to the minimum. Could you please recommend me rclone options to keep low the memory consumption by rclone?

Best,
Raymond

ncw · January 7, 2021, 7:40pm

What is your usage? Reading? Writing? Or both?

How big are the files?

I suspect this is a problem with the Azure backend, but we can find out for sure. Can you follow the instructions here and produce a memory profile after the memory has got large?

Raimondas · January 7, 2021, 9:20pm

Files are from 16Kb and up to 500Mb each. Average - 2-3Mb each.
Usage - both reading and writing. Usually - reading. Writing in most cases - only one time.
I'll try to produce a memory profile tomorrow. I cannot unmount the storage now.

Raimondas · January 8, 2021, 4:13pm

system is running for 2 hours.
According to the htop, rclone took about 8Gb of RAM.
1533383 root 20 0 8733M 7970M 31492 S 15.3 24.8 33:15.72 rclone mount --log-f...

rclone mount --log-file=/var/log/dcmsys/rclone.log --log-level ERROR --config=/etc/rclone.conf --allow-other --default-permissions --max-read-ahead=4M --vfs-cache-mode=writes --vfs-cache-max-size=1G --azureblob-memory-pool-flush-time=1m0s --buffer-size=0 --azureblob-memory-pool-use-mmap --use-mmap --dir-cache-time=2h --timeout=1h --cache-dir=/var/dcmsys/cache --uid=1100 --gid=1100 --cache-chunk-no-memory --rc --size-only --allow-non-empty blob:studiesh /var/dcmsys/storages/GCP

I have file pprof.rclone.alloc_objects.alloc_space.inuse_objects.inuse_space.026.pb.gz but not sure how to share it here.

Fetching profile over HTTP from http://localhost:5572/debug/pprof/heap
Saved profile in /root/pprof/pprof.rclone.alloc_objects.alloc_space.inuse_objects.inuse_space.026.pb.gz
File: rclone
Type: inuse_space
Time: Jan 8, 2021 at 11:10am (EST)
Showing nodes accounting for 3758.28MB, 96.70% of 3886.48MB total
Dropped 131 nodes (cum <= 19.43MB)
      flat  flat%   sum%        cum   cum%
 3605.21MB 92.76% 92.76%  3605.21MB 92.76%  github.com/Azure/azure-storage-blob-go/azblob.copyFromReader.func1
   40.51MB  1.04% 93.81%    40.51MB  1.04%  github.com/Azure/azure-storage-blob-go/azblob.BlobGetPropertiesResponse.NewMetadata
      25MB  0.64% 94.45%       25MB  0.64%  github.com/rclone/rclone/vfs.newFile
   24.01MB  0.62% 95.07%    29.01MB  0.75%  github.com/rclone/rclone/vfs/vfscache.newItem
      20MB  0.51% 95.58%       20MB  0.51%  strings.(*Builder).grow (inline)
      14MB  0.36% 95.94%    23.71MB  0.61%  context.WithDeadline
   12.50MB  0.32% 96.26%  3327.12MB 85.61%  github.com/rclone/rclone/backend/azureblob.(*Fs).Put
       8MB  0.21% 96.47%    26.79MB  0.69%  github.com/rclone/rclone/vfs.newRWFileHandle
    4.04MB   0.1% 96.57%    21.04MB  0.54%  github.com/rclone/rclone/vfs.(*Dir)._readDirFromEntries
       2MB 0.051% 96.62%   100.30MB  2.58%  github.com/rclone/rclone/cmd/mount.(*Dir).Create
    1.50MB 0.039% 96.66%    26.22MB  0.67%  github.com/Azure/azure-storage-blob-go/azblob.(*SharedKeyCredential).New.func1
       1MB 0.026% 96.69%    30.01MB  0.77%  github.com/rclone/rclone/vfs/vfscache.(*Cache)._get
    0.50MB 0.013% 96.70%    24.53MB  0.63%  github.com/rclone/rclone/cmd/mount.(*Dir).Lookup
         0     0% 96.70%   137.96MB  3.55%  bazil.org/fuse/fs.(*Server).Serve.func1
         0     0% 96.70%   137.96MB  3.55%  bazil.org/fuse/fs.(*Server).handleRequest
         0     0% 96.70%   137.96MB  3.55%  bazil.org/fuse/fs.(*Server).serve
         0     0% 96.70%    23.71MB  0.61%  context.WithTimeout
         0     0% 96.70%    50.43MB  1.30%  github.com/Azure/azure-pipeline-go/pipeline.(*pipeline).Do
         0     0% 96.70%    50.43MB  1.30%  github.com/Azure/azure-pipeline-go/pipeline.PolicyFunc.Do
         0     0% 96.70%  3606.21MB 92.79%  github.com/Azure/azure-storage-blob-go/azblob.(*copier).sendChunk
         0     0% 96.70%    28.50MB  0.73%  github.com/Azure/azure-storage-blob-go/azblob.BlobURL.GetProperties
         0     0% 96.70%    50.43MB  1.30%  github.com/Azure/azure-storage-blob-go/azblob.NewRetryPolicyFactory.func1.1
         0     0% 96.70%    50.43MB  1.30%  github.com/Azure/azure-storage-blob-go/azblob.NewTelemetryPolicyFactory.func1.1
         0     0% 96.70%    50.43MB  1.30%  github.com/Azure/azure-storage-blob-go/azblob.NewUniqueRequestIDPolicyFactory.func1.1
         0     0% 96.70%  3611.92MB 92.94%  github.com/Azure/azure-storage-blob-go/azblob.UploadStreamToBlockBlob
         0     0% 96.70%    28.50MB  0.73%  github.com/Azure/azure-storage-blob-go/azblob.blobClient.GetProperties
         0     0% 96.70%  3611.92MB 92.94%  github.com/Azure/azure-storage-blob-go/azblob.copyFromReader
         0     0% 96.70%    24.71MB  0.64%  github.com/Azure/azure-storage-blob-go/azblob.responderPolicy.Do
         0     0% 96.70%       20MB  0.51%  github.com/rclone/rclone/backend/azureblob.(*Fs).list
         0     0% 96.70%  3686.24MB 94.85%  github.com/rclone/rclone/backend/azureblob.(*Object).Update
         0     0% 96.70%  3611.92MB 92.94%  github.com/rclone/rclone/backend/azureblob.(*Object).Update.func1
         0     0% 96.70%    46.01MB  1.18%  github.com/rclone/rclone/backend/azureblob.(*Object).decodeMetaDataFromPropertiesResponse
         0     0% 96.70%    74.51MB  1.92%  github.com/rclone/rclone/backend/azureblob.(*Object).readMetaData
         0     0% 96.70%    28.50MB  0.73%  github.com/rclone/rclone/backend/azureblob.(*Object).readMetaData.func1
         0     0% 96.70%  3652.22MB 93.97%  github.com/rclone/rclone/fs.pacerInvoker
         0     0% 96.70%  3702.75MB 95.27%  github.com/rclone/rclone/fs/operations.Copy
         0     0% 96.70%       41MB  1.06%  github.com/rclone/rclone/lib/pacer.(*Pacer).Call
         0     0% 96.70%  3611.21MB 92.92%  github.com/rclone/rclone/lib/pacer.(*Pacer).CallNoRetry (inline)
         0     0% 96.70%  3652.22MB 93.97%  github.com/rclone/rclone/lib/pacer.(*Pacer).call
         0     0% 96.70%    21.50MB  0.55%  github.com/rclone/rclone/vfs.(*Dir).Create
         0     0% 96.70%    24.03MB  0.62%  github.com/rclone/rclone/vfs.(*Dir).Stat
         0     0% 96.70%    29.54MB  0.76%  github.com/rclone/rclone/vfs.(*Dir)._readDir
         0     0% 96.70%    24.03MB  0.62%  github.com/rclone/rclone/vfs.(*Dir).stat
         0     0% 96.70%    76.80MB  1.98%  github.com/rclone/rclone/vfs.(*File).Open
         0     0% 96.70%       20MB  0.51%  github.com/rclone/rclone/vfs.(*File).Path
         0     0% 96.70%    26.79MB  0.69%  github.com/rclone/rclone/vfs.(*File).openRW
         0     0% 96.70%    30.01MB  0.77%  github.com/rclone/rclone/vfs/vfscache.(*Cache).Exists
         0     0% 96.70%    30.01MB  0.77%  github.com/rclone/rclone/vfs/vfscache.(*Cache).get
         0     0% 96.70%  3694.24MB 95.05%  github.com/rclone/rclone/vfs/vfscache.(*Item).Close.func2
         0     0% 96.70%  3694.24MB 95.05%  github.com/rclone/rclone/vfs/vfscache.(*Item)._store
         0     0% 96.70%  3694.24MB 95.05%  github.com/rclone/rclone/vfs/vfscache.(*Item).store
         0     0% 96.70%  3694.24MB 95.05%  github.com/rclone/rclone/vfs/vfscache/writeback.(*WriteBack).upload
         0     0% 96.70%       20MB  0.51%  path.Join
         0     0% 96.70%       20MB  0.51%  strings.(*Builder).Grow (inline)
         0     0% 96.70%       20MB  0.51%  strings.Join
         0     0% 96.70%  3613.04MB 92.96%  sync.(*Pool).Get

Best,
Raymond

ncw · January 8, 2021, 11:03pm

Hmm, the leak is in the Azure blob SDK.

I have a feeling that this is fixed in the SDK version in use in the beta.

Was that from the beta?

Maybe I forgot to merge something.

Raimondas · January 8, 2021, 11:35pm

I used rclone version:
rclone v1.54.0-beta.5040.71edc75ca

ncw · January 9, 2021, 12:28pm

I dug into this again and remembered I made an issue about this on the azure blob SDK

dcendents · January 9, 2021, 3:12pm

@Raimondas Until the bug @ncw mentioned in the Azure blob SDK is fixed I suggest you stick with rclone 1.52.3, that version works fine with low memory usage (can't say for sure about CPU but I think the high usage might be related to garbage collection).

See my original post about the problem here, I think it is the same as yours: Rclone 1.53.3 memory (leak?) problem

Raimondas · January 9, 2021, 5:33pm

Hi Daniel,
You saved the day! Thanks a lot for your advice!
I installed rclone v1.52.3 and after 1h45m I see rclone has eaten only 450Mb. That's success!

Saved profile in /root/pprof/pprof.rclone.alloc_objects.alloc_space.inuse_objects.inuse_space.035.pb.gz
File: rclone
Type: inuse_space
Time: Jan 9, 2021 at 12:33pm (EST)
Showing nodes accounting for 138.46MB, 98.19% of 141.01MB total
Dropped 25 nodes (cum <= 0.71MB)
      flat  flat%   sum%        cum   cum%
   20.50MB 14.54% 14.54%    20.50MB 14.54%  strings.(*Builder).grow (inline)
      18MB 12.77% 27.31%       18MB 12.77%  github.com/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.BlobGetPropertiesResponse.NewMetadata
   13.50MB  9.58% 36.88%    13.50MB  9.58%  github.com/rclone/rclone/vfs.newFile
      12MB  8.51% 45.40%       12MB  8.51%  github.com/rclone/rclone/vfs.newDir
    8.81MB  6.24% 51.64%     8.81MB  6.24%  github.com/rclone/rclone/vendor/bazil.org/fuse/fs.(*Server).saveNode
       7MB  4.96% 56.60%    46.01MB 32.63%  github.com/rclone/rclone/backend/azureblob.(*Fs).Put
       6MB  4.26% 60.86%        6MB  4.26%  time.AfterFunc
       6MB  4.26% 65.12%        6MB  4.26%  net/textproto.(*Reader).ReadMIMEHeader
    5.50MB  3.90% 69.02%    11.50MB  8.16%  context.WithDeadline
    5.26MB  3.73% 72.74%     5.26MB  3.73%  github.com/rclone/rclone/vfs.(*Dir).addObject
    4.54MB  3.22% 75.96%    17.54MB 12.44%  github.com/rclone/rclone/vfs.(*Dir)._readDirFromEntries
    4.50MB  3.19% 79.16%     4.50MB  3.19%  os.newFile
       4MB  2.84% 81.99%        4MB  2.84%  github.com/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.uploadStream.func1
       4MB  2.84% 84.83%        4MB  2.84%  context.(*cancelCtx).Done
       4MB  2.84% 87.67%        4MB  2.84%  encoding/xml.copyValue
    3.50MB  2.48% 90.15%    27.76MB 19.69%  github.com/rclone/rclone/vfs.newRWFileHandle
       3MB  2.13% 92.28%        3MB  2.13%  github.com/rclone/rclone/fs.NewDir (inline)
    2.50MB  1.77% 94.05%     3.64MB  2.58%  github.com/rclone/rclone/vendor/bazil.org/fuse.(*Conn).ReadRequest
    1.50MB  1.06% 95.11%     1.50MB  1.06%  github.com/rclone/rclone/vendor/github.com/aws/aws-sdk-go/aws/endpoints.init
    1.50MB  1.06% 96.18%     1.50MB  1.06%  github.com/rclone/rclone/vfs.(*File).addWriter
    1.20MB  0.85% 97.03%     1.20MB  0.85%  time.startTimer
    1.14MB  0.81% 97.84%     1.14MB  0.81%  github.com/rclone/rclone/vendor/bazil.org/fuse.allocMessage
    0.50MB  0.35% 98.19%     1.10MB  0.78%  time.NewTimer
         0     0% 98.19%    11.50MB  8.16%  context.WithTimeout
         0     0% 98.19%     1.05MB  0.74%  crypto/tls.(*Conn).Handshake
         0     0% 98.19%     1.05MB  0.74%  crypto/tls.(*Conn).clientHandshake
         0     0% 98.19%     1.05MB  0.74%  crypto/tls.(*clientHandshakeState).doFullHandshake
         0     0% 98.19%     1.05MB  0.74%  crypto/tls.(*clientHandshakeState).handshake
         0     0% 98.19%        4MB  2.84%  encoding/xml.(*Decoder).Decode (inline)
         0     0% 98.19%        4MB  2.84%  encoding/xml.(*Decoder).DecodeElement
         0     0% 98.19%        4MB  2.84%  encoding/xml.(*Decoder).unmarshal
         0     0% 98.19%        4MB  2.84%  encoding/xml.(*Decoder).unmarshalPath
         0     0% 98.19%        4MB  2.84%  encoding/xml.Unmarshal
         0     0% 98.19%        7MB  4.96%  github.com/rclone/rclone/backend/azureblob.(*Fs).List
         0     0% 98.19%        3MB  2.13%  github.com/rclone/rclone/backend/azureblob.(*Fs).itemToDirEntry
         0     0% 98.19%        7MB  4.96%  github.com/rclone/rclone/backend/azureblob.(*Fs).list
         0     0% 98.19%        4MB  2.84%  github.com/rclone/rclone/backend/azureblob.(*Fs).list.func1
         0     0% 98.19%        7MB  4.96%  github.com/rclone/rclone/backend/azureblob.(*Fs).listDir
         0     0% 98.19%        3MB  2.13%  github.com/rclone/rclone/backend/azureblob.(*Fs).listDir.func1
         0     0% 98.19%    39.01MB 27.66%  github.com/rclone/rclone/backend/azureblob.(*Object).Update
         0     0% 98.19%        5MB  3.55%  github.com/rclone/rclone/backend/azureblob.(*Object).Update.func1
         0     0% 98.19%    18.50MB 13.12%  github.com/rclone/rclone/backend/azureblob.(*Object).decodeMetaDataFromPropertiesResponse
         0     0% 98.19%    34.01MB 24.12%  github.com/rclone/rclone/backend/azureblob.(*Object).readMetaData
         0     0% 98.19%    16.10MB 11.42%  github.com/rclone/rclone/backend/azureblob.(*Object).readMetaData.func1
         0     0% 98.19%     5.10MB  3.62%  github.com/rclone/rclone/backend/azureblob.httpClientFactory.func1.1
         0     0% 98.19%    40.26MB 28.55%  github.com/rclone/rclone/cmd/mount.(*Dir).Create
         0     0% 98.19%    19.54MB 13.86%  github.com/rclone/rclone/cmd/mount.(*Dir).Lookup
         0     0% 98.19%        1MB  0.71%  github.com/rclone/rclone/cmd/mount.(*Dir).ReadDirAll
         0     0% 98.19%    49.51MB 35.11%  github.com/rclone/rclone/cmd/mount.(*FileHandle).Flush
         0     0% 98.19%     3.64MB  2.58%  github.com/rclone/rclone/cmd/mount.mount.func1
         0     0% 98.19%    25.10MB 17.80%  github.com/rclone/rclone/fs.pacerInvoker
         0     0% 98.19%     5.10MB  3.62%  github.com/rclone/rclone/fs/fshttp.(*Transport).RoundTrip
         0     0% 98.19%        3MB  2.13%  github.com/rclone/rclone/fs/list.DirSorted
         0     0% 98.19%    46.01MB 32.63%  github.com/rclone/rclone/fs/operations.Copy
         0     0% 98.19%    20.10MB 14.26%  github.com/rclone/rclone/lib/pacer.(*Pacer).Call
         0     0% 98.19%        5MB  3.55%  github.com/rclone/rclone/lib/pacer.(*Pacer).CallNoRetry (inline)
         0     0% 98.19%    25.10MB 17.80%  github.com/rclone/rclone/lib/pacer.(*Pacer).call
         0     0% 98.19%     1.14MB  0.81%  github.com/rclone/rclone/vendor/bazil.org/fuse.getMessage (inline)
         0     0% 98.19%     3.64MB  2.58%  github.com/rclone/rclone/vendor/bazil.org/fuse/fs.(*Server).Serve
         0     0% 98.19%   119.12MB 84.48%  github.com/rclone/rclone/vendor/bazil.org/fuse/fs.(*Server).Serve.func1
         0     0% 98.19%   119.12MB 84.48%  github.com/rclone/rclone/vendor/bazil.org/fuse/fs.(*Server).handleRequest
         0     0% 98.19%     8.81MB  6.24%  github.com/rclone/rclone/vendor/bazil.org/fuse/fs.(*Server).saveLookup
         0     0% 98.19%   119.12MB 84.48%  github.com/rclone/rclone/vendor/bazil.org/fuse/fs.(*Server).serve
         0     0% 98.19%    20.60MB 14.61%  github.com/rclone/rclone/vendor/github.com/Azure/azure-pipeline-go/pipeline.(*pipeline).Do
         0     0% 98.19%    20.60MB 14.61%  github.com/rclone/rclone/vendor/github.com/Azure/azure-pipeline-go/pipeline.PolicyFunc.Do
         0     0% 98.19%     9.10MB  6.45%  github.com/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.(*SharedKeyCredential).New.func1
         0     0% 98.19%    16.10MB 11.42%  github.com/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.BlobURL.GetProperties
         0     0% 98.19%        4MB  2.84%  github.com/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.ContainerURL.ListBlobsHierarchySegment
         0     0% 98.19%     5.10MB  3.62%  github.com/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.NewRequestLogPolicyFactory.func1.1
         0     0% 98.19%    20.60MB 14.61%  github.com/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.NewRetryPolicyFactory.func1.1
         0     0% 98.19%    20.60MB 14.61%  github.com/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.NewTelemetryPolicyFactory.func1.1
         0     0% 98.19%    20.60MB 14.61%  github.com/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.NewUniqueRequestIDPolicyFactory.func1.1
         0     0% 98.19%        5MB  3.55%  github.com/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.UploadStreamToBlockBlob
         0     0% 98.19%    16.10MB 11.42%  github.com/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.blobClient.GetProperties
         0     0% 98.19%        4MB  2.84%  github.com/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.containerClient.ListBlobHierarchySegment
         0     0% 98.19%        4MB  2.84%  github.com/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.containerClient.listBlobHierarchySegmentResponder
         0     0% 98.19%     9.10MB  6.45%  github.com/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.responderPolicy.Do
         0     0% 98.19%        5MB  3.55%  github.com/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.uploadStream
         0     0% 98.19%    12.50MB  8.87%  github.com/rclone/rclone/vfs.(*Dir).Create
         0     0% 98.19%        1MB  0.71%  github.com/rclone/rclone/vfs.(*Dir).ReadDirAll
         0     0% 98.19%    19.54MB 13.86%  github.com/rclone/rclone/vfs.(*Dir).Stat
         0     0% 98.19%    20.54MB 14.57%  github.com/rclone/rclone/vfs.(*Dir)._readDir
         0     0% 98.19%    19.54MB 13.86%  github.com/rclone/rclone/vfs.(*Dir).stat
         0     0% 98.19%    27.76MB 19.69%  github.com/rclone/rclone/vfs.(*File).Open
         0     0% 98.19%        9MB  6.38%  github.com/rclone/rclone/vfs.(*File).Path
         0     0% 98.19%    27.76MB 19.69%  github.com/rclone/rclone/vfs.(*File).openRW
         0     0% 98.19%    11.50MB  8.16%  github.com/rclone/rclone/vfs.(*File).osPath
         0     0% 98.19%    49.51MB 35.11%  github.com/rclone/rclone/vfs.(*RWFileHandle).Flush
         0     0% 98.19%    49.51MB 35.11%  github.com/rclone/rclone/vfs.(*RWFileHandle).flushWrites
         0     0% 98.19%    21.26MB 15.08%  github.com/rclone/rclone/vfs.(*RWFileHandle).openPending
         0     0% 98.19%    42.01MB 29.79%  github.com/rclone/rclone/vfs/vfscache.(*Cache).Store
         0     0% 98.19%    11.50MB  8.16%  github.com/rclone/rclone/vfs/vfscache.(*Cache).ToOSPath (inline)
         0     0% 98.19%    42.01MB 29.79%  github.com/rclone/rclone/vfs/vfscache.copyObj
         0     0% 98.19%     5.10MB  3.62%  net/http.(*Client).Do (inline)
         0     0% 98.19%     5.10MB  3.62%  net/http.(*Client).do
         0     0% 98.19%     5.10MB  3.62%  net/http.(*Client).send
         0     0% 98.19%     5.10MB  3.62%  net/http.(*Transport).RoundTrip (inline)
         0     0% 98.19%     5.10MB  3.62%  net/http.(*Transport).roundTrip
         0     0% 98.19%     1.05MB  0.74%  net/http.(*persistConn).addTLS.func2
         0     0% 98.19%        6MB  4.26%  net/http.(*persistConn).readLoop
         0     0% 98.19%        6MB  4.26%  net/http.(*persistConn).readResponse
         0     0% 98.19%     1.10MB  0.78%  net/http.(*persistConn).roundTrip
         0     0% 98.19%        6MB  4.26%  net/http.ReadResponse
         0     0% 98.19%     5.10MB  3.62%  net/http.send
         0     0% 98.19%     4.50MB  3.19%  os.OpenFile
         0     0% 98.19%     4.50MB  3.19%  os.openFileNolog
         0     0% 98.19%        9MB  6.38%  path.Join
         0     0% 98.19%    11.50MB  8.16%  path/filepath.Join (inline)
         0     0% 98.19%    11.50MB  8.16%  path/filepath.join
         0     0% 98.19%     1.50MB  1.06%  runtime.doInit
         0     0% 98.19%     1.50MB  1.06%  runtime.main
         0     0% 98.19%    20.50MB 14.54%  strings.(*Builder).Grow (inline)
         0     0% 98.19%    20.50MB 14.54%  strings.Join
         0     0% 98.19%     1.05MB  0.74%  sync.(*Once).Do (inline)
         0     0% 98.19%     1.05MB  0.74%  sync.(*Once).doSlow
         0     0% 98.19%     1.14MB  0.81%  sync.(*Pool).Get

Best,
Raymond

Raimondas · January 11, 2021, 2:18pm

update.
rclone v1.52.3

rclone mount --log-file=/var/log/dcmsys/rclone.log --log-level ERROR --config=/etc/rclone.conf --allow-other --default-permissions --max-read-ahead=4M --vfs-cache-mode=writes --vfs-cache-max-size=1G --azureblob-memory-pool-flush-time=1m0s --buffer-size=0 --azureblob-memory-pool-use-mmap --use-mmap --dir-cache-time=2h --timeout=1h --cache-dir=/var/dcmsys/cache --uid=1100 --gid=1100 --cache-chunk-no-memory --rc --size-only --allow-non-empty blob:studiesh /var/dcmsys/storages/GCP

after 2 days of running, htop shows that rclone uses about 9Gb or ram:

1860391 root 20 0 9.8G 9210M 10612 S 0.0 28.6 18h41:38 rclone mount --log-file=/var/log/dcmsys/rclone.log --log-level ERROR --config=/etc/rcl

Here is the memory profile:
> Fetching profile over HTTP from localhost:5572/debug/pprof/heap
> Saved profile in /root/pprof/pprof.rclone.alloc_objects.alloc_space.inuse_objects.inuse_space.037.pb.gz
> File: rclone
> Type: inuse_space
> Time: Jan 11, 2021 at 9:10am (EST)
> Showing nodes accounting for 4060.02MB, 97.81% of 4151MB total
> Dropped 99 nodes (cum <= 20.75MB)
>       flat  flat%   sum%        cum   cum%
>   712.17MB 17.16% 17.16%   712.17MB 17.16%  github.c/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.BlobGetPropertiesResponse.NewMetadata
>   674.10MB 16.24% 33.40%   674.10MB 16.24%  strings.(*Builder).grow (inline)
>   521.61MB 12.57% 45.96%   521.61MB 12.57%  github.c/rclone/rclone/vfs.newFile
>   270.02MB  6.50% 52.47%   487.07MB 11.73%  context.WithDeadline
>   258.01MB  6.22% 58.68%   258.01MB  6.22%  net/textproto.(*Reader).ReadMIMEHeader
>   245.52MB  5.91% 64.60%   245.52MB  5.91%  context.(*cancelCtx).Done
>   242.03MB  5.83% 70.43%  1852.36MB 44.62%  github.c/rclone/rclone/backend/azureblob.(*Fs).Put
>   232.52MB  5.60% 76.03%   232.52MB  5.60%  os.newFile
>   180.51MB  4.35% 80.38%   198.55MB  4.78%  time.AfterFunc
>   161.33MB  3.89% 84.26%   161.33MB  3.89%  github.c/rclone/rclone/vfs.(*Dir).addObject
>   146.51MB  3.53% 87.79%   964.89MB 23.24%  github.c/rclone/rclone/vfs.newRWFileHandle
>   141.01MB  3.40% 91.19%   153.53MB  3.70%  github.c/rclone/rclone/vendor/bazil.org/fuse.(*Conn).ReadRequest
>   128.04MB  3.08% 94.28%   128.04MB  3.08%  github.c/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.uploadStream.func1
>       64MB  1.54% 95.82%       64MB  1.54%  encoding/base64.(*Encoding).EncodeToString
>    40.50MB  0.98% 96.79%    40.50MB  0.98%  github.c/rclone/rclone/vfs.(*File).addWriter
>    23.58MB  0.57% 97.36%    23.58MB  0.57%  time.startTimer
>    11.50MB  0.28% 97.64%  1496.51MB 36.05%  github.c/rclone/rclone/cmd/mount.(*Dir).Create
>     7.06MB  0.17% 97.81%    25.06MB   0.6%  github.c/rclone/rclone/vfs.(*Dir)._readDirFromEntries
>          0     0% 97.81%   487.07MB 11.73%  context.WithTimeout
>          0     0% 97.81%       21MB  0.51%  encoding/xml.(*Decoder).Decode (inline)
>          0     0% 97.81%       21MB  0.51%  encoding/xml.(*Decoder).DecodeElement
>          0     0% 97.81%       21MB  0.51%  encoding/xml.(*Decoder).unmarshal
>          0     0% 97.81%       21MB  0.51%  encoding/xml.(*Decoder).unmarshalPath
>          0     0% 97.81%       21MB  0.51%  encoding/xml.Unmarshal
>          0     0% 97.81%       27MB  0.65%  github.c/rclone/rclone/backend/azureblob.(*Fs).List
>          0     0% 97.81%       37MB  0.89%  github.c/rclone/rclone/backend/azureblob.(*Fs).list
>          0     0% 97.81%    27.50MB  0.66%  github.c/rclone/rclone/backend/azureblob.(*Fs).list.func1
>          0     0% 97.81%       27MB  0.65%  github.c/rclone/rclone/backend/azureblob.(*Fs).listDir
>          0     0% 97.81%  1623.84MB 39.12%  github.c/rclone/rclone/backend/azureblob.(*Object).Update
>          0     0% 97.81%   224.08MB  5.40%  github.c/rclone/rclone/backend/azureblob.(*Object).Update.func1
>          0     0% 97.81%   774.68MB 18.66%  github.c/rclone/rclone/backend/azureblob.(*Object).decodeMetaDataFromPropertiesResponse
>          0     0% 97.81%  1399.23MB 33.71%  github.c/rclone/rclone/backend/azureblob.(*Object).readMetaData
>          0     0% 97.81%   624.05MB 15.03%  github.c/rclone/rclone/backend/azureblob.(*Object).readMetaData.func1
>          0     0% 97.81%   228.52MB  5.51%  github.c/rclone/rclone/backend/azureblob.httpClientFactory.func1.1
>          0     0% 97.81%    35.56MB  0.86%  github.c/rclone/rclone/cmd/mount.(*Dir).Lookup
>          0     0% 97.81%  1923.88MB 46.35%  github.c/rclone/rclone/cmd/mount.(*FileHandle).Flush
>          0     0% 97.81%   153.53MB  3.70%  github.c/rclone/rclone/cmd/mount.mount.func1
>          0     0% 97.81%   876.14MB 21.11%  github.c/rclone/rclone/fs.pacerInvoker
>          0     0% 97.81%   228.52MB  5.51%  github.c/rclone/rclone/fs/fshttp.(*Transport).RoundTrip
>          0     0% 97.81%  1853.87MB 44.66%  github.c/rclone/rclone/fs/operations.Copy
>          0     0% 97.81%   651.56MB 15.70%  github.c/rclone/rclone/lib/pacer.(*Pacer).Call
>          0     0% 97.81%   223.58MB  5.39%  github.c/rclone/rclone/lib/pacer.(*Pacer).CallNoRetry (inline)
>          0     0% 97.81%   876.14MB 21.11%  github.c/rclone/rclone/lib/pacer.(*Pacer).call
>          0     0% 97.81%   153.53MB  3.70%  github.c/rclone/rclone/vendor/bazil.org/fuse/fs.(*Server).Serve
>          0     0% 97.81%  3459.25MB 83.34%  github.c/rclone/rclone/vendor/bazil.org/fuse/fs.(*Server).Serve.func1
>          0     0% 97.81%  3477.28MB 83.77%  github.c/rclone/rclone/vendor/bazil.org/fuse/fs.(*Server).handleRequest
>          0     0% 97.81%  3477.28MB 83.77%  github.c/rclone/rclone/vendor/bazil.org/fuse/fs.(*Server).serve
>          0     0% 97.81%   737.59MB 17.77%  github.c/rclone/rclone/vendor/github.com/Azure/azure-pipeline-go/pipeline.(*pipeline).Do
>          0     0% 97.81%   737.59MB 17.77%  github.c/rclone/rclone/vendor/github.c/Azure/azure-pipeline-go/pipeline.PolicyFunc.Do
>          0     0% 97.81%   250.02MB  6.02%  github.c/rclone/rclone/vendor/github.c/Azure/azure-storage-blob-go/azblob.(*SharedKeyCredential).New.func1
>          0     0% 97.81%    83.54MB  2.01%  github.c/rclone/rclone/vendor/github.c/Azure/azure-storage-blob-go/azblob.(*uploadStreamToBlockBlobOptions).end
>          0     0% 97.81%   624.05MB 15.03%  github.c/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.BlobURL.GetProperties
>          0     0% 97.81%    83.54MB  2.01%  github.c/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.BlockBlobURL.Upload
>          0     0% 97.81%    27.50MB  0.66%  github.c/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.ContainerURL.ListBlobsHierarchySegment
>          0     0% 97.81%   228.52MB  5.51%  github.c/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.NewRequestLogPolicyFactory.func1.1
>          0     0% 97.81%   737.59MB 17.77%  github.c/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.NewRetryPolicyFactory.func1.1
>          0     0% 97.81%   737.59MB 17.77%  github.c/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.NewTelemetryPolicyFactory.func1.1
>          0     0% 97.81%   737.59MB 17.77%  github./rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.NewUniqueRequestIDPolicyFactory.func1.1
>          0     0% 97.81%   224.08MB  5.40%  github./rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.UploadStreamToBlockBlob
>          0     0% 97.81%   624.05MB 15.03%  github./rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.blobClient.GetProperties
>          0     0% 97.81%    83.54MB  2.01%  github./rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.blockBlobClient.Upload
>          0     0% 97.81%    27.50MB  0.66%  github./rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.containerClient.ListBlobHierarchySegment
>          0     0% 97.81%       21MB  0.51%  github./rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.containerClient.listBlobHierarchySegmentResponder
>          0     0% 97.81%   249.52MB  6.01%  github./rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.responderPolicy.Do
>          0     0% 97.81%   224.08MB  5.40%  github./rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.uploadStream
>          0     0% 97.81%   520.11MB 12.53%  github./rclone/rclone/vfs.(*Dir).Create
>          0     0% 97.81%    35.56MB  0.86%  github./rclone/rclone/vfs.(*Dir).Stat
>          0     0% 97.81%    39.56MB  0.95%  github.c/rclone/rclone/vfs.(*Dir)._readDir
>          0     0% 97.81%    35.56MB  0.86%  github./rclone/rclone/vfs.(*Dir).stat
>          0     0% 97.81%   964.89MB 23.24%  github./rclone/rclone/vfs.(*File).Open
>          0     0% 97.81%   282.03MB  6.79%  github./rclone/rclone/vfs.(*File).Path
>          0     0% 97.81%   964.89MB 23.24%  github./rclone/rclone/vfs.(*File).openRW
>          0     0% 97.81%   389.56MB  9.38%  github/rclone/rclone/vfs.(*File).osPath
>          0     0% 97.81%  1923.88MB 46.35%  github./rclone/rclone/vfs.(*RWFileHandle).Flush
>          0     0% 97.81%  1923.88MB 46.35%  github.c/rclone/rclone/vfs.(*RWFileHandle).flushWrites
>          0     0% 97.81%   781.89MB 18.84%  github/rclone/rclone/vfs.(*RWFileHandle).openPending
>          0     0% 97.81%  1641.85MB 39.55%  github.c/rclone/rclone/vfs/vfscache.(*Cache).Store
>          0     0% 97.81%   389.56MB  9.38%  github.c/rclone/rclone/vfs/vfscache.(*Cache).ToOSPath (inline)
>          0     0% 97.81%  1641.85MB 39.55%  github.c/rclone/rclone/vfs/vfscache.copyObj
>          0     0% 97.81%   228.52MB  5.51%  net/http.(*Client).Do (inline)
>          0     0% 97.81%   228.52MB  5.51%  net/http.(*Client).do
>          0     0% 97.81%   228.52MB  5.51%  net/http.(*Client).send
>          0     0% 97.81%   228.52MB  5.51%  net/http.(*Transport).RoundTrip (inline)
>          0     0% 97.81%   228.52MB  5.51%  net/http.(*Transport).roundTrip
>          0     0% 97.81%   258.01MB  6.22%  net/http.(*persistConn).readLoop
>          0     0% 97.81%   258.01MB  6.22%  net/http.(*persistConn).readResponse
>          0     0% 97.81%   258.01MB  6.22%  net/http.ReadResponse
>          0     0% 97.81%   228.52MB  5.51%  net/http.send
>          0     0% 97.81%   232.52MB  5.60%  os.OpenFile
>          0     0% 97.81%   232.52MB  5.60%  os.openFileNolog
>          0     0% 97.81%   284.03MB  6.84%  path.Join
>          0     0% 97.81%   389.56MB  9.38%  path/filepath.Join (inline)
>          0     0% 97.81%   389.56MB  9.38%  path/filepath.join
>          0     0% 97.81%   674.10MB 16.24%  strings.(*Builder).Grow (inline)
>          0     0% 97.81%   673.60MB 16.23%  strings.Join

Best,
Raymond

ncw · January 11, 2021, 4:26pm

Would you mind trying this beta or later?

This has had the large file upload code reworked to use the facilities from the SDK entirely. I think this may help, but I'm not 100% sure!

Raimondas · January 13, 2021, 9:17am

Uptime 3h:

rclone v1.54.0-beta.5060.e0b5a13a1

os/arch: linux/amd64
go version: go1.15.6

htop:
2703199 root 20 0 10.6G 9745M 31376 S 24.6 30.3 40:39.95 rclone mount --log-file=/var/log/dcmsys/rclone.log --log-level ERROR --config=/etc/rclone.conf --allow-other --default-permissions --max-read-ahead=4M --vfs-cache-mode=writes --vfs-cache-max-size=1G --azureblob-memory-pool-f

Saved profile in /root/pprof/pprof.rclone.alloc_objects.alloc_space.inuse_objects.inuse_space.041.pb.gz
File: rclone
Type: inuse_space
Time: Jan 13, 2021 at 4:15am (EST)
Showing nodes accounting for 4827.99MB, 96.60% of 4997.81MB total
Dropped 147 nodes (cum <= 24.99MB)
      flat  flat%   sum%        cum   cum%
 4601.54MB 92.07% 92.07%  4601.54MB 92.07%  github.com/Azure/azure-storage-blob-go/azblob.copyFromReader.func1
   58.01MB  1.16% 93.23%    58.01MB  1.16%  github.com/Azure/azure-storage-blob-go/azblob.BlobGetPropertiesResponse.NewMetadata
   36.01MB  0.72% 93.95%    46.01MB  0.92%  github.com/rclone/rclone/vfs/vfscache.newItem
   34.01MB  0.68% 94.63%    34.01MB  0.68%  github.com/rclone/rclone/vfs.newFile
   27.33MB  0.55% 95.18%    27.33MB  0.55%  bazil.org/fuse.allocMessage
   22.50MB  0.45% 95.63%    31.77MB  0.64%  context.WithDeadline
   15.50MB  0.31% 95.94%  4735.57MB 94.75%  github.com/rclone/rclone/backend/azureblob.(*Fs).Put
   13.50MB  0.27% 96.21%    26.74MB  0.53%  github.com/rclone/rclone/vfs.newRWFileHandle
    9.50MB  0.19% 96.40%    36.83MB  0.74%  bazil.org/fuse.(*Conn).ReadRequest
    5.09MB   0.1% 96.50%    27.09MB  0.54%  github.com/rclone/rclone/vfs.(*Dir)._readDirFromEntries
       2MB  0.04% 96.54%    38.55MB  0.77%  github.com/Azure/azure-storage-blob-go/azblob.(*SharedKeyCredential).New.func1
    1.50MB  0.03% 96.57%   129.75MB  2.60%  github.com/rclone/rclone/cmd/mount.(*Dir).Create
    0.50MB  0.01% 96.58%  4610.32MB 92.25%  github.com/Azure/azure-storage-blob-go/azblob.copyFromReader
    0.50MB  0.01% 96.59%  4610.82MB 92.26%  github.com/Azure/azure-storage-blob-go/azblob.UploadStreamToBlockBlob
    0.50MB  0.01% 96.60%    70.82MB  1.42%  github.com/Azure/azure-storage-blob-go/azblob.NewRetryPolicyFactory.func1.1
         0     0% 96.60%    27.33MB  0.55%  bazil.org/fuse.getMessage (inline)
         0     0% 96.60%    36.83MB  0.74%  bazil.org/fuse/fs.(*Server).Serve
         0     0% 96.60%   176.84MB  3.54%  bazil.org/fuse/fs.(*Server).Serve.func1
         0     0% 96.60%   176.84MB  3.54%  bazil.org/fuse/fs.(*Server).handleRequest
         0     0% 96.60%   176.84MB  3.54%  bazil.org/fuse/fs.(*Server).serve
         0     0% 96.60%    31.77MB  0.64%  context.WithTimeout
         0     0% 96.60%    70.82MB  1.42%  github.com/Azure/azure-pipeline-go/pipeline.(*pipeline).Do
         0     0% 96.60%    70.82MB  1.42%  github.com/Azure/azure-pipeline-go/pipeline.PolicyFunc.Do
         0     0% 96.60%  4602.04MB 92.08%  github.com/Azure/azure-storage-blob-go/azblob.(*copier).sendChunk
         0     0% 96.60%    47.50MB  0.95%  github.com/Azure/azure-storage-blob-go/azblob.BlobURL.GetProperties
         0     0% 96.60%    25.54MB  0.51%  github.com/Azure/azure-storage-blob-go/azblob.NewRequestLogPolicyFactory.func1.1
         0     0% 96.60%    70.82MB  1.42%  github.com/Azure/azure-storage-blob-go/azblob.NewTelemetryPolicyFactory.func1.1
         0     0% 96.60%    70.82MB  1.42%  github.com/Azure/azure-storage-blob-go/azblob.NewUniqueRequestIDPolicyFactory.func1.1
         0     0% 96.60%    47.50MB  0.95%  github.com/Azure/azure-storage-blob-go/azblob.blobClient.GetProperties
         0     0% 96.60%    36.04MB  0.72%  github.com/Azure/azure-storage-blob-go/azblob.responderPolicy.Do
         0     0% 96.60%  4722.57MB 94.49%  github.com/rclone/rclone/backend/azureblob.(*Object).Update
         0     0% 96.60%  4610.82MB 92.26%  github.com/rclone/rclone/backend/azureblob.(*Object).Update.func1
         0     0% 96.60%    64.51MB  1.29%  github.com/rclone/rclone/backend/azureblob.(*Object).decodeMetaDataFromPropertiesResponse
         0     0% 96.60%   112.02MB  2.24%  github.com/rclone/rclone/backend/azureblob.(*Object).readMetaData
         0     0% 96.60%    47.50MB  0.95%  github.com/rclone/rclone/backend/azureblob.(*Object).readMetaData.func1
         0     0% 96.60%    25.54MB  0.51%  github.com/rclone/rclone/backend/azureblob.httpClientFactory.func1.1
         0     0% 96.60%    35.86MB  0.72%  github.com/rclone/rclone/cmd/mount.(*Dir).Lookup
         0     0% 96.60%    36.83MB  0.74%  github.com/rclone/rclone/cmd/mount.mount.func2
         0     0% 96.60%  4669.32MB 93.43%  github.com/rclone/rclone/fs.pacerInvoker
         0     0% 96.60%  4738.57MB 94.81%  github.com/rclone/rclone/fs/operations.Copy
         0     0% 96.60%    58.78MB  1.18%  github.com/rclone/rclone/lib/pacer.(*Pacer).Call
         0     0% 96.60%  4609.55MB 92.23%  github.com/rclone/rclone/lib/pacer.(*Pacer).CallNoRetry (inline)
         0     0% 96.60%  4669.32MB 93.43%  github.com/rclone/rclone/lib/pacer.(*Pacer).call
         0     0% 96.60%    33.01MB  0.66%  github.com/rclone/rclone/vfs.(*Dir).Create
         0     0% 96.60%    35.86MB  0.72%  github.com/rclone/rclone/vfs.(*Dir).Stat
         0     0% 96.60%    37.86MB  0.76%  github.com/rclone/rclone/vfs.(*Dir)._readDir
         0     0% 96.60%    35.86MB  0.72%  github.com/rclone/rclone/vfs.(*Dir).stat
         0     0% 96.60%    95.75MB  1.92%  github.com/rclone/rclone/vfs.(*File).Open
         0     0% 96.60%    26.74MB  0.53%  github.com/rclone/rclone/vfs.(*File).openRW
         0     0% 96.60%    46.01MB  0.92%  github.com/rclone/rclone/vfs/vfscache.(*Cache).Exists
         0     0% 96.60%    46.01MB  0.92%  github.com/rclone/rclone/vfs/vfscache.(*Cache)._get
         0     0% 96.60%    46.01MB  0.92%  github.com/rclone/rclone/vfs/vfscache.(*Cache).get
         0     0% 96.60%  4720.57MB 94.45%  github.com/rclone/rclone/vfs/vfscache.(*Item).Close.func2
         0     0% 96.60%  4720.57MB 94.45%  github.com/rclone/rclone/vfs/vfscache.(*Item)._store
         0     0% 96.60%  4720.57MB 94.45%  github.com/rclone/rclone/vfs/vfscache.(*Item).store
         0     0% 96.60%  4720.57MB 94.45%  github.com/rclone/rclone/vfs/vfscache/writeback.(*WriteBack).upload
         0     0% 96.60%  4628.87MB 92.62%  sync.(*Pool).Get

Raimondas · January 13, 2021, 9:18am

for comparison:

rclone version
rclone v1.52.3

uptime 3 days

htop:
1860391 root 20 0 12.7G 12.0G 9228 S 25.4 38.1 29h42:23 rclone mount --log-file=/var/log/dcmsys/rclone.log --log-level ERROR --config=/etc/rclone.conf --allow-other --default-permissions --max-read-ahead=4M --vfs-cache-mode=writes --vfs-cache-max-size=1G --azureblob-memory-pool-f

Fetching profile over HTTP from http://localhost:5572/debug/pprof/heap
Saved profile in /root/pprof/pprof.rclone.alloc_objects.alloc_space.inuse_objects.inuse_space.038.pb.gz
File: rclone
Type: inuse_space
Time: Jan 13, 2021 at 1:21am (EST)
Showing nodes accounting for 5.62GB, 98.02% of 5.73GB total
Dropped 98 nodes (cum <= 0.03GB)
      flat  flat%   sum%        cum   cum%
    0.95GB 16.58% 16.58%     0.95GB 16.58%  github.com/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.BlobGetPropertiesResponse.NewMetadata
    0.90GB 15.62% 32.20%     0.90GB 15.62%  strings.(*Builder).grow (inline)
    0.68GB 11.91% 44.12%     0.68GB 11.91%  github.com/rclone/rclone/vfs.newFile
    0.37GB  6.40% 50.52%     0.65GB 11.29%  context.WithDeadline
    0.34GB  5.95% 56.46%     0.34GB  5.95%  net/textproto.(*Reader).ReadMIMEHeader
    0.33GB  5.81% 62.28%     0.33GB  5.81%  context.(*cancelCtx).Done
    0.31GB  5.45% 67.73%     0.31GB  5.45%  github.com/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.uploadStream.func1
    0.31GB  5.42% 73.15%     2.60GB 45.39%  github.com/rclone/rclone/backend/azureblob.(*Fs).Put
    0.31GB  5.36% 78.51%     0.31GB  5.36%  os.newFile
    0.25GB  4.39% 82.90%     0.26GB  4.51%  time.AfterFunc
    0.22GB  3.86% 86.75%     0.22GB  3.86%  github.com/rclone/rclone/vfs.(*Dir).addObject
    0.19GB  3.40% 90.15%     1.29GB 22.52%  github.com/rclone/rclone/vfs.newRWFileHandle
    0.19GB  3.38% 93.54%     0.25GB  4.42%  github.com/rclone/rclone/vendor/bazil.org/fuse.(*Conn).ReadRequest
    0.09GB  1.54% 95.08%     0.09GB  1.54%  encoding/base64.(*Encoding).EncodeToString
    0.06GB  1.04% 96.12%     0.06GB  1.04%  github.com/rclone/rclone/vendor/bazil.org/fuse.allocMessage
    0.06GB  1.02% 97.14%     0.06GB  1.02%  github.com/rclone/rclone/vfs.(*File).addWriter
    0.03GB  0.56% 97.70%     0.03GB  0.56%  time.startTimer
    0.02GB  0.32% 98.02%     1.99GB 34.70%  github.com/rclone/rclone/cmd/mount.(*Dir).Create
         0     0% 98.02%     0.65GB 11.29%  context.WithTimeout
         0     0% 98.02%     0.04GB  0.63%  encoding/xml.(*Decoder).Decode (inline)
         0     0% 98.02%     0.04GB  0.63%  encoding/xml.(*Decoder).DecodeElement
         0     0% 98.02%     0.04GB  0.63%  encoding/xml.(*Decoder).unmarshal
         0     0% 98.02%     0.04GB  0.63%  encoding/xml.(*Decoder).unmarshalPath
         0     0% 98.02%     0.04GB  0.63%  encoding/xml.Unmarshal
         0     0% 98.02%     0.03GB  0.57%  github.com/rclone/rclone/backend/azureblob.(*Fs).List
         0     0% 98.02%     0.05GB  0.95%  github.com/rclone/rclone/backend/azureblob.(*Fs).list
         0     0% 98.02%     0.04GB  0.76%  github.com/rclone/rclone/backend/azureblob.(*Fs).list.func1
         0     0% 98.02%     0.03GB   0.6%  github.com/rclone/rclone/backend/azureblob.(*Fs).listDir
         0     0% 98.02%     2.32GB 40.53%  github.com/rclone/rclone/backend/azureblob.(*Object).Update
         0     0% 98.02%     0.45GB  7.82%  github.com/rclone/rclone/backend/azureblob.(*Object).Update.func1
         0     0% 98.02%     1.04GB 18.08%  github.com/rclone/rclone/backend/azureblob.(*Object).decodeMetaDataFromPropertiesResponse
         0     0% 98.02%     1.88GB 32.83%  github.com/rclone/rclone/backend/azureblob.(*Object).readMetaData
         0     0% 98.02%     0.85GB 14.90%  github.com/rclone/rclone/backend/azureblob.(*Object).readMetaData.func1
         0     0% 98.02%     0.33GB  5.83%  github.com/rclone/rclone/backend/azureblob.httpClientFactory.func1.1
         0     0% 98.02%     0.04GB  0.64%  github.com/rclone/rclone/cmd/mount.(*Dir).Lookup
         0     0% 98.02%     2.70GB 47.05%  github.com/rclone/rclone/cmd/mount.(*FileHandle).Flush
         0     0% 98.02%     0.25GB  4.42%  github.com/rclone/rclone/cmd/mount.mount.func1
         0     0% 98.02%     1.34GB 23.37%  github.com/rclone/rclone/fs.pacerInvoker
         0     0% 98.02%     0.33GB  5.83%  github.com/rclone/rclone/fs/fshttp.(*Transport).RoundTrip
         0     0% 98.02%     2.60GB 45.38%  github.com/rclone/rclone/fs/operations.Copy
         0     0% 98.02%     0.90GB 15.67%  github.com/rclone/rclone/lib/pacer.(*Pacer).Call
         0     0% 98.02%     0.44GB  7.65%  github.com/rclone/rclone/lib/pacer.(*Pacer).CallNoRetry (inline)
         0     0% 98.02%     1.34GB 23.37%  github.com/rclone/rclone/lib/pacer.(*Pacer).call
         0     0% 98.02%     0.06GB  1.04%  github.com/rclone/rclone/vendor/bazil.org/fuse.getMessage (inline)
         0     0% 98.02%     0.25GB  4.42%  github.com/rclone/rclone/vendor/bazil.org/fuse/fs.(*Server).Serve
         0     0% 98.02%     4.75GB 82.86%  github.com/rclone/rclone/vendor/bazil.org/fuse/fs.(*Server).Serve.func1
         0     0% 98.02%     4.75GB 82.86%  github.com/rclone/rclone/vendor/bazil.org/fuse/fs.(*Server).handleRequest
         0     0% 98.02%     4.75GB 82.86%  github.com/rclone/rclone/vendor/bazil.org/fuse/fs.(*Server).serve
         0     0% 98.02%     1.02GB 17.77%  github.com/rclone/rclone/vendor/github.com/Azure/azure-pipeline-go/pipeline.(*pipeline).Do
         0     0% 98.02%     1.02GB 17.77%  github.com/rclone/rclone/vendor/github.com/Azure/azure-pipeline-go/pipeline.PolicyFunc.Do
         0     0% 98.02%     0.37GB  6.48%  github.com/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.(*SharedKeyCredential).New.func1
         0     0% 98.02%     0.11GB  1.99%  github.com/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.(*uploadStreamToBlockBlobOptions).end
         0     0% 98.02%     0.85GB 14.90%  github.com/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.BlobURL.GetProperties
         0     0% 98.02%     0.11GB  1.98%  github.com/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.BlockBlobURL.Upload
         0     0% 98.02%     0.04GB  0.76%  github.com/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.ContainerURL.ListBlobsHierarchySegment
         0     0% 98.02%     0.33GB  5.83%  github.com/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.NewRequestLogPolicyFactory.func1.1
         0     0% 98.02%     1.02GB 17.77%  github.com/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.NewRetryPolicyFactory.func1.1
         0     0% 98.02%     1.02GB 17.77%  github.com/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.NewTelemetryPolicyFactory.func1.1
         0     0% 98.02%     1.02GB 17.77%  github.com/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.NewUniqueRequestIDPolicyFactory.func1.1
         0     0% 98.02%     0.45GB  7.82%  github.com/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.UploadStreamToBlockBlob
         0     0% 98.02%     0.85GB 14.90%  github.com/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.blobClient.GetProperties
         0     0% 98.02%     0.11GB  1.98%  github.com/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.blockBlobClient.Upload
         0     0% 98.02%     0.04GB  0.76%  github.com/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.containerClient.ListBlobHierarchySegment
         0     0% 98.02%     0.04GB  0.63%  github.com/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.containerClient.listBlobHierarchySegmentResponder
         0     0% 98.02%     0.37GB  6.46%  github.com/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.responderPolicy.Do
         0     0% 98.02%     0.45GB  7.82%  github.com/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.uploadStream
         0     0% 98.02%     0.68GB 11.85%  github.com/rclone/rclone/vfs.(*Dir).Create
         0     0% 98.02%     0.04GB  0.64%  github.com/rclone/rclone/vfs.(*Dir).Stat
         0     0% 98.02%     0.04GB  0.75%  github.com/rclone/rclone/vfs.(*Dir)._readDir
         0     0% 98.02%     0.04GB  0.64%  github.com/rclone/rclone/vfs.(*Dir).stat
         0     0% 98.02%     1.29GB 22.52%  github.com/rclone/rclone/vfs.(*File).Open
         0     0% 98.02%     0.37GB  6.52%  github.com/rclone/rclone/vfs.(*File).Path
         0     0% 98.02%     1.29GB 22.52%  github.com/rclone/rclone/vfs.(*File).openRW
         0     0% 98.02%     0.52GB  9.04%  github.com/rclone/rclone/vfs.(*File).osPath
         0     0% 98.02%     2.70GB 47.05%  github.com/rclone/rclone/vfs.(*RWFileHandle).Flush
         0     0% 98.02%     2.70GB 47.05%  github.com/rclone/rclone/vfs.(*RWFileHandle).flushWrites
         0     0% 98.02%     1.04GB 18.17%  github.com/rclone/rclone/vfs.(*RWFileHandle).openPending
         0     0% 98.02%     2.32GB 40.53%  github.com/rclone/rclone/vfs/vfscache.(*Cache).Store
         0     0% 98.02%     0.52GB  9.04%  github.com/rclone/rclone/vfs/vfscache.(*Cache).ToOSPath (inline)
         0     0% 98.02%     2.32GB 40.53%  github.com/rclone/rclone/vfs/vfscache.copyObj
         0     0% 98.02%     0.33GB  5.83%  net/http.(*Client).Do (inline)
         0     0% 98.02%     0.33GB  5.83%  net/http.(*Client).do
         0     0% 98.02%     0.33GB  5.83%  net/http.(*Client).send
         0     0% 98.02%     0.33GB  5.83%  net/http.(*Transport).RoundTrip (inline)
         0     0% 98.02%     0.33GB  5.83%  net/http.(*Transport).roundTrip
         0     0% 98.02%     0.34GB  5.95%  net/http.(*persistConn).readLoop
         0     0% 98.02%     0.34GB  5.95%  net/http.(*persistConn).readResponse
         0     0% 98.02%     0.34GB  5.95%  net/http.ReadResponse
         0     0% 98.02%     0.33GB  5.83%  net/http.send
         0     0% 98.02%     0.31GB  5.36%  os.OpenFile
         0     0% 98.02%     0.31GB  5.36%  os.openFileNolog
         0     0% 98.02%     0.38GB  6.55%  path.Join
         0     0% 98.02%     0.52GB  9.04%  path/filepath.Join (inline)
         0     0% 98.02%     0.52GB  9.04%  path/filepath.join
         0     0% 98.02%     0.90GB 15.62%  strings.(*Builder).Grow (inline)
         0     0% 98.02%     0.89GB 15.59%  strings.Join
         0     0% 98.02%     0.06GB  1.04%  sync.(*Pool).Get

ncw · January 13, 2021, 10:54am

Can you try this one?

v1.54.0-beta.5061.705d00b8b.fix-azureblob-memory on branch fix-azureblob-memory (uploaded in 15-30 mins)

This contains one of the potential fixes to the azureblob SDK. That fix isn't merged yet but it looks good to me.

Ask for settings recommendation for azureblob

What is the problem you are having with rclone?

What is your rclone version (output from rclone version)

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Which cloud storage system are you using? (eg Google Drive)

The command you were trying to run (eg rclone copy /tmp remote:tmp)

The rclone config contents with secrets removed.

What is your rclone version (output from `rclone version`)

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)