Sync task takes 24 hours to run against a AWS S3 bucket to Backblaze B2 Cloud Storage. The source S3 bucket is 4TB large with lots and lots of small files (approx 16 million). Any additional flags or strategy to speed up the sync so it does not take 24 hours?
Running on a dedicated EC2 m5.large instance so 2 vCPUs and 8GB of memory so lot's of resources to throw at it.
- os/version: ubuntu 18.04 (64 bit)
- os/kernel: 5.4.0-1056-aws (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.16.8
- go/linking: static
- go/tags: none
AWS S3 and Backblaze B2 Cloud Storage
rclone --log-file=/var/log/rclone --log-level INFO sync --use-mmap --buffer-size=16M --checkers=128 --transfers 32 s3:acme-inc-cloud-storage crypt: