Hi Team,
We have recently installed rclone on our server to manage uploads, downloads, and deletions between Maximo and COS buckets on syd01. Below are the details of our configuration and observed performance:
Server Resources
free -h
total used free shared buff/cache available
Mem: 31G 9.2G 5.3G 207M 16G 21G
Swap: 2.0G 479M 1.5G
Initial Rclone Mount Configuration
rclone mount remote:mmis-doclinks-prod /mmisfilestore
--daemon
--config /root/.config/rclone/rclone.conf
--log-file=/var/log/rclone.log
--allow-other
--cache-dir /disk1/mmisfilestore_cache
--vfs-cache-mode full
--s3-acl=private
--s3-upload-cutoff=64Mi
--s3-chunk-size=64Mi
--s3-upload-concurrency=1
--multi-thread-streams=4
--multi-thread-cutoff=16Mi
--buffer-size=16Mi
--transfers 16
--s3-disable-checksum=true
Observed Upload Latency:
1 file (5.03 MB) → 1m 22s
2.9 MB → 2 min
5 MB → 2.5 min
5 MB → 1m 7s
18 files (4.49 MB total) → 2m 12s
3 files (11.5 MB total) → 4m 17s
Updated Rclone Mount Configuration
rclone mount remote:mmis-doclinks-prod /mmisfilestore
--daemon
--config /root/.config/rclone/rclone.conf
--log-file=/var/log/rclone.log
--log-level INFO
--allow-other
--umask 002
--dir-cache-time 10m
--poll-interval 5m
--cache-dir /disk1/mmisfilestore_cache
--vfs-cache-mode full
--vfs-cache-max-size 15G
--vfs-cache-max-age 2h
--buffer-size 128M
--transfers 32
--checkers 16
--s3-upload-concurrency 8
--s3-chunk-size 32M
--s3-disable-checksum
--s3-no-check-bucket
--no-modtime
--use-server-modtime
--multi-thread-streams 4
--multi-thread-cutoff 32M
Observed Upload Latency:
1 file (20 MB) → 4.35 sec
1 file (5 MB) → 1.4 sec
3 files (11.5 MB total) → 3m 8s
5 MB → 2.5 min
4 MB → 40 sec
5 MB → 1m 20s
9 MB → 2 min
7 MB → 1m 30s
1 file (5.03 MB) → 1m 25s
3 MB → 40 sec
Observations
Upload times for similar file sizes are inconsistent
We have ~70 concurrent users and 32 GB RAM
Current mount configuration may need further tuning for production
Could you please help us optimize these configurations? We aim to move this setup to production as soon as possible.
Thanks & regards,
Tejaswini L