Hello
I am using using rclone to ease the acess of my computer in other city by using mount so there is increased compatibility when opening files.
When using filezilla/winscp it transfers at 2.4-2.5MB/s (max upload) while mount transfers at 1.9-2.0MB/s.
I tested with a .bat the time of the copy command too via SFTP, it took 1.2x times longer like mount did.
This is only a problem of SFTP transfers as I tested mounting the same setup on FTP and it uses my full upload,2.4-2.5MB/s. I rather not be using unsafe FTP tho.
The only parameter I use is --vfs-cache-mode minimal, I looked at the options and global options it looked to me reasonable.
SFTP uses 1.85% (of 4 threads on a 2 core second gen mobile i5, 1 transfer)
FTP uses 0.13%
Ok I looked deeper on the options and I noticed --transfers is only for each file. So the solution of the previous problem of SFTP it being slower still persists because it doesn’t open more connections on a single file.
Pratical test. on a 205MB file. (mebibytes like every value I measured)
I don’t know how to measure speed so I timed with windows’ batch.
filezilla over SFTP takes between 87 and 88s(looking at the timer), by process monitor writes at a median 2.38MB/s. ’rclone copy’ over FTP takes 86.98s total, retired couple of times between 86s and 87s, median 2.38MB/s, avg 2.356MB/s ’rclone copy’ over SFTP takes 99.18s total, sometimes between 99s and 103s. avg 2.05MB/s
same file split in 14MB , 15 files.
over SFTP
(106.49s total) --transfers 1
(92.04s total) --transfers 2
(89.43s total) --transfers 4
(88.65s total) --transfers 8 (88.57s total) --transfers 16 (increased max ssshd connections on the server)
FTP
(91.25s total) --transfers 1
(87.16s total) --transfers 2 (86.38s total) --transfers 3
more than --transfers 8 sometimes gives error that it failed to open a socket
Yes on multiple files multiple connections maybe saves the speed.
But filezilla transfers at FTP speed without opening multiple connections, I looked at the filezilla connections via process explorer and only used 1.
by the number of receives they sure have a different transfer algorithm. Looks like filezilla requests multiple at once?
I can’t speculate further without looking at the rclone’s SFTP code.
Using windows explorer and copy pasting when it is mounted as far as I can tell transfers 1 file at a time so --transfers doesn’t help.
I expect that FileZilla does overlapping reads to get the performance up or uses bigger buffers.
I had a quick look at the code for the SFTP library - it appears that it does use overlapping reads if the read buffer is big enough.
The default read buffer is 32k - it is possible to increase that, however not all servers are guaranteed to use more than 32k. I wonder if FileZilla does that? I looked at FileZilla’s source briefly then I remembered why I’m not a C++ programmer any more
Rclone uses 1 MB buffers to accept the reads from SFTP…
It might be worth experimenting with the --buffer-size parameter and see what different that makes. Try 0 as a baseline too.
