How to reach max speed Dwn/UP on s3 compatible storage?

[Surfparadise (Marco)]

Hi

I try to achieve the max performance in download/upload from/to S3 compatible object storage.

I have available 4 Gb/s network connection and a block volume of 900MB/s capable (180GB RAM and 12 CPU Cores), but I reach only 360MB/s in read and around 280MB/s in download.

The command that I use is this one:

rclone.exe copy --s3-upload-concurrency 128 --s3-chunk-size 128M --progress remote:bucket D:\test.dat

rclone v1.59.2

• os/version: Microsoft Windows Server 2016 Standard 1607 (64 bit)
• os/kernel: 10.0.14393.4770 (x86_64)
• os/type: windows
• os/arch: amd64
• go/version: go1.18.6
• go/linking: static
• go/tags: cmount

Could you help me please?
Thanks in advance

[ncw (Nick Craig-Wood)]

Are you copying a single large file or lots of files? If lots of files, then increase --transfers.

If a single large file, then investigate this parameter

 --multi-thread-streams int             Max number of streams to use for multi-thread downloads (default 4)

Increasing --checkers will increase concurrency generally.

@Surfparadise

--s3-upload-concurrency 128

Note that this only takes effect copying to s3, not from s3 like you are doing in your example.

[ncw (Nick Craig-Wood)]

For uploading large files, investigate

  --s3-chunk-size SizeSuffix             Chunk size to use for uploading (default 5Mi)
 --s3-disable-checksum                  Don't store MD5 checksum with object metadata
  --s3-upload-concurrency int            Concurrency for multipart uploads (default 4)

Also try this - we've seen big speedups with this recently

  --s3-disable-http2                     Disable usage of http2 for S3 backends

[Ole (Ole Frost)]

Hi Marco,

I suggest you carefully read all the posts in this recent thread:

Best way to maximize 100gbps GCP gVNIC?

What is the problem you are having with rclone? I want to perform Cloud-to-Cloud transfers using rclone that maximize the bandwidth I can get (100gbps) I am currently seeing only 2.2gbps speeds when doing a GCS to crypt(GDrive) transfer. Even Ookla's speedtest CLI reports higher (albeit not close to 100gbps) Run the command 'rclone version' and share the full output of the command. rclone v1.53.3-DEV - os/arch: linux/amd64 - go version: go1.18 Which cloud storage system are you using? (e…

In it you will find an entire walkthrough of a systematic approach to tune rclone up/download parameters.

You may not need to tune the S3 specific parameters once the general optimization is completed, the bottlenecks are typically in more general areas (think --transfers and --checkers).

@ncw

Also try this - we've seen big speedups with this recently

  --s3-disable-http2                     Disable usage of http2 for S3 backends

This was one of the learnings from the thread linked above

[Surfparadise (Marco)]

The best performance that I can reach in upload doesn't exceed 370/380MB/s
The download it's more problematic .... start above 300MB/s and after when ETA go to 0s the speed decrease around 0/b. The hdd is reading (???? why normally has to write...)...
At the end the download consume the double of time than the upload.....

[Ole (Ole Frost)]

Seems like you missed my point. Please take a look at this post (from me):
Best way to maximize 100gbps GCP gVNIC? - #6 by Ole

What would the corresponding command be in your situation? and what speed does it show? does it change over time?

What speed and latency do you see when downloading from speedtest.net?

Also please post the redacted output from

rclone config show yourRemote: