SFTP 80% slower (on 16ms ping)

I am using using rclone to ease the acess of my computer in other city by using mount so there is increased compatibility when opening files.

When using filezilla/winscp it transfers at 2.4-2.5MB/s (max upload) while mount transfers at 1.9-2.0MB/s.
I tested with a .bat the time of the copy command too via SFTP, it took 1.2x times longer like mount did.

This is only a problem of SFTP transfers as I tested mounting the same setup on FTP and it uses my full upload,2.4-2.5MB/s. I rather not be using unsafe FTP tho.

I looked up and SFTP speed and ping problems should have been fixed here https://github.com/ncw/rclone/issues/1158

So I don’t know why SFTP is stil slower in rclone.

Also would FTPS be at full speed like FTP? I wonder if this is going to ever be merged.

What sort of speed do you get if you just do an rclone copy rather than copy files from the mount?

What parameters are you using with rclone mount?

How much CPU is rclone using during the transfer?

FTPS should in theory be as fast as FTP…

The only parameter I use is --vfs-cache-mode minimal, I looked at the options and global options it looked to me reasonable.

SFTP uses 1.85% (of 4 threads on a 2 core second gen mobile i5, 1 transfer)
FTP uses 0.13%

Ok I looked deeper on the options and I noticed --transfers is only for each file. So the solution of the previous problem of SFTP it being slower still persists because it doesn’t open more connections on a single file.

Pratical test. on a 205MB file. (mebibytes like every value I measured)
I don’t know how to measure speed so I timed with windows’ batch.

filezilla over SFTP takes between 87 and 88s(looking at the timer), by process monitor writes at a median 2.38MB/s.
’rclone copy’ over FTP takes 86.98s total, retired couple of times between 86s and 87s, median 2.38MB/s, avg 2.356MB/s
’rclone copy’ over SFTP takes 99.18s total, sometimes between 99s and 103s. avg 2.05MB/s

same file split in 14MB , 15 files.
over SFTP
(106.49s total) --transfers 1
(92.04s total) --transfers 2
(89.43s total) --transfers 4
(88.65s total) --transfers 8
(88.57s total) --transfers 16 (increased max ssshd connections on the server)

(91.25s total) --transfers 1
(87.16s total) --transfers 2
(86.38s total) --transfers 3
more than --transfers 8 sometimes gives error that it failed to open a socket

Yes on multiple files multiple connections maybe saves the speed.
But filezilla transfers at FTP speed without opening multiple connections, I looked at the filezilla connections via process explorer and only used 1.

by the number of receives they sure have a different transfer algorithm. Looks like filezilla requests multiple at once?
I can’t speculate further without looking at the rclone’s SFTP code.

Using windows explorer and copy pasting when it is mounted as far as I can tell transfers 1 file at a time so --transfers doesn’t help.

That is very interesting analysis - thanks.

I expect that FileZilla does overlapping reads to get the performance up or uses bigger buffers.

I had a quick look at the code for the SFTP library - it appears that it does use overlapping reads if the read buffer is big enough.

The default read buffer is 32k - it is possible to increase that, however not all servers are guaranteed to use more than 32k. I wonder if FileZilla does that? I looked at FileZilla’s source briefly then I remembered why I’m not a C++ programmer any more :wink:

Rclone uses 1 MB buffers to accept the reads from SFTP…

It might be worth experimenting with the --buffer-size parameter and see what different that makes. Try 0 as a baseline too.