I try to achieve the max performance in download/upload from/to S3 compatible object storage.
I have available 4 Gb/s network connection and a block volume of 900MB/s capable (180GB RAM and 12 CPU Cores), but I reach only 360MB/s in read and around 280MB/s in download.
The command that I use is this one:
rclone.exe copy --s3-upload-concurrency 128 --s3-chunk-size 128M --progress remote:bucket D:\test.dat
os/version: Microsoft Windows Server 2016 Standard 1607 (64 bit)
os/kernel: 10.0.14393.4770 (x86_64)
Could you help me please?
Thanks in advance
Are you copying a single large file or lots of files? If lots of files, then increase
If a single large file, then investigate this parameter
--multi-thread-streams int Max number of streams to use for multi-thread downloads (default 4)
--checkers will increase concurrency generally.
Note that this only takes effect copying
to s3, not from s3 like you are doing in your example.
For uploading large files, investigate
--s3-chunk-size SizeSuffix Chunk size to use for uploading (default 5Mi)
--s3-disable-checksum Don't store MD5 checksum with object metadata
--s3-upload-concurrency int Concurrency for multipart uploads (default 4)
Also try this - we've seen big speedups with this recently
--s3-disable-http2 Disable usage of http2 for S3 backends
I suggest you carefully read all the posts in this recent thread:
What is the problem you are having with rclone?
I want to perform Cloud-to-Cloud transfers using rclone that maximize the bandwidth I can get (100gbps)
I am currently seeing only 2.2gbps speeds when doing a GCS to crypt(GDrive) transfer. Even Ookla's speedtest CLI reports higher (albeit not close to 100gbps)
Run the command 'rclone version' and share the full output of the command.
- os/arch: linux/amd64
- go version: go1.18
Which cloud storage system are you using? (e…
In it you will find an entire walkthrough of a systematic approach to tune rclone up/download parameters.
You may not need to tune the S3 specific parameters once the general optimization is completed, the bottlenecks are typically in more general areas (think --transfers and --checkers).
This was one of the learnings from the thread linked above
The best performance that I can reach in upload doesn't exceed 370/380MB/s
The download it's more problematic .... start above 300MB/s and after when ETA go to 0s the speed decrease around 0/b. The hdd is reading (???? why normally has to write...)...
At the end the download consume the double of time than the upload.....
Seems like you missed my point. Please take a look at this post (from me):
Best way to maximize 100gbps GCP gVNIC? - #6 by Ole
What would the corresponding command be in your situation? and what speed does it show? does it change over time?
What speed and latency do you see when downloading from
Also please post the redacted output from
rclone config show yourRemote: