I'm trying to implement a high load system with cloud based storage.
What is the problem you are having with rclone?
currently I have a VM system with 32Gb ram on board and 4 CPU cores. RAM and CPU consumption of rclone is way to high. rclone consumed 18Gb of RAM after 6h of working.
CPU is jumping from 20 to 100%.
Could you please give me an advice what settings should I use for mounting azure blob in order to keep CPU and RAM for rclone on mininum, or at least - 10Gb max RAM.
I can pay some money for help (tuning the system/config) if you exactly know what to do, or you are the rclone developer.
What is your rclone version (output from
Which OS you are using and how many bits (eg Windows 7, 64 bit)
CentOS 8.2.2004 (4.18.0-193.el8.x86_64)
Which cloud storage system are you using? (eg Google Drive)
Azure blob storage
The command you were trying to run (eg
rclone copy /tmp remote:tmp)
I am using following line to mount disk:
rclone mount --log-file=/var/log/dcmsys/rclone.log --log-level ERROR --config=/etc/rclone.conf --allow-other --default-permissions --max-read-ahead=16M --vfs-cache-mode=writes --vfs-cache-max-size=10G --azureblob-memory-pool-use-mmap --use-mmap --cache-chunk-no-memory --cache-dir=/var/dcmsys/cache --uid=1100 --gid=1100 --allow-non-empty blob:studiesh /var/dcmsys/storages/GCP
next time I'll try following config for fstab:
rclonefs#blob:studiesh /var/dcmsys/storages/GCP fuse _netdev,config=/etc/rclone.conf,allow-other,default-permissions,rw,max-read-ahead=16M,vfs-cache-mode=writes,vfs-cache-max-size=2G,azureblob-memory-pool-flush-time=1m0s,buffer-size=0,azureblob-memory-pool-use-mmap,use-mmap,cache-chunk-no-memory,cache-db-purge,buffer-size=32M,dir-cache-time=72h,timeout=1h,cache-dir=/var/dcmsys/cache,uid=dcmsys,gid=dcmsys,size-only,allow-non-empty 0 0
The rclone config contents with secrets removed.
[blob] type = azureblob account = az1vna01sa key =