That is a fantastic bit of testing - thank you.
Increasing concurrency seems to be a consistent win up to 32 or 64.
However increasing chunk size seems to be a net loss which is not what I expected.
From that table I don't see any reason to increase the default chunk size of 4M.
I could increase the default concurrency though..
I did a few tests myself on a cloud VM and it seems that the performance increase starts off quite linear when increasing concurrency. The memory usage goes up by approx chunk_size * concurrency as you might expect.
chunk_size 4MiB
| concurrency | speed (MiB/s) |
|---|---|
| 4 | 8.7 |
| 8 | 13 |
| 16 | 28 |
| 32 | 56 |
| 64 | 113 |
| 128 | 170 |
We could certainly afford to increase the default from 4 - I could make it 16 which would make the memory usage per transfer be 64M which is in line with the other rclone backends.
What performance do you see for chunk size 4M with concurrency 4,8,16,32?