Hi @ncw and thank you very much for this very thoughtful work and reply. Definitely interesting information to have, I hadn't considered the "actual" backend (ie sftp source) as having extra flags to change.
Indeed, speed wise, I have seen from 12-15MiB/s for single file transfers between our remote system, using tools such as rsync or scp; just an example.
And so far yes, --transfers works well, but we are still hitting that sort of "limit", as mentioned.
Here's a bunch of testing I've done, let me know if anything's missing.
START
Test file is 5.2 GB (5,209,574,759 bytes)
Local: openssh-server/oldstable,now 1:7.9p1-10+deb10u2
Local (alt SSD system): openssh 8.9p1-1
Remote: openssh 8.9p1-1
Command
~/rclone159BetaSFTP/rclone -vv --progress --sftp-set-modtime=true --create-empty-src-dirs copy BIG_TEST.mp4 crypt-largestor:test/
Test 1
--sftp-chunk-size=252k --sftp-concurrency=64
Transferred: 4.853 GiB / 4.853 GiB, 100%, 2.246 MiB/s, ETA 0s
Transferred: 1 / 1, 100%
Elapsed time: 19m10.9s
Actual average transfer speed was 4-5MiB/s, inline with iotop
$ dstat -n 5
Says average of 18-19M/s with Send speed
$ sudo iotop
Reports 4-5M/s
Test 2
--sftp-chunk-size=32k --sftp-concurrency=512
Transferred: 4.853 GiB / 4.853 GiB, 100%, 5.985 MiB/s, ETA 0s
Transferred: 1 / 1, 100%
Elapsed time: 7m19.8s
Actual average transfer speed was 11-12MiB/s, inline with iotop
$ dstat -n 5
Says average of 18-19M/s with Send speed
$ sudo iotop
Reports 11-12M/s
So it looks like chunk size of 32k is a good choice.
Testing increased --sftp-concurrency with --sftp-chunk-size=32k, same single file and other flags, worked as such:
--sftp-concurrency=1536 , 11.8MiB/s average
--sftp-concurrency=2048 , 11.5MiB/s average
--sftp-concurrency=2560 , 11.8MiB/s average
Next, testing with multiple transfers... produces disappointing speeds. All of the files have sizes above 5GiB.
--sftp-chunk-size=32k --sftp-concurrency=512 --transfers=8
36-40 MiB/s transfer speed; dstat says 60-70M/s send speed
--sftp-chunk-size=32k --sftp-concurrency=1536 --transfers=8
36-40 MiB/s transfer speed; dstat says 60-70M/s send speed
--sftp-chunk-size=252k --sftp-concurrency=128 --transfers=8
15-16MiB/s transfer speed; dstat says 65-70M/s send speed
--sftp-chunk-size=252k --sftp-concurrency=1024 --transfers=8
16-17 MiB/s transfer speed; dstat says 55-60M/s send speed
For all of those, iotop reports speeds mimicing rclone Transfer speed
Increasing multiple file transfers doesn't improve transfer speed, except dstat continues to climb at or near 1Gbit/s speeds.
Also, oddly,
with beta build, --sftp-concurrency=64 --transfers=16; did not use --sftp-chunk-size
36-40 MiB/s transfer speed; dstat says 65-70M/s send
And again moving back to my non-beta rclone, speeds are 65-70MiB/s, with 12-24 transfers. dstat reports accurate send speeds too.
END
So this is definitely some odd results... any further suggestions? I'm curious why dstat would report higher numbers than what's actually being sent as well.
Also hopefully the single test file I used was acceptable size wise.