One general thought before I get to my problem: rclone must be the most discouraging project that I've ever encountered when it comes to raising issues. I have literally forced myself to create this post after giving up on GitHub
I'm trying to switch my file sync to
rclone with FTP backend (FastMail) but it seems to be extremely slow. FastMail provides WebDav backend as well which is at least 5-6x faster than FTP however lack of modification time synchronization in WebDav is kind of a blocker.
$ rclone version rclone 1.57.0 - os/version: pld 3.0 - os/kernel: 5.15.6-1 (armv7l) - os/type: linux - os/arch: arm - go/version: go1.16.8 - go/linking: dynamic - go/tags: none
rclone --transfers 2 -P /dir remote:/dir
[remote] type = ftp host = ftp.fastmail.com user = user pass = pass disable_epsv = true
rclone -vv let me share my suspicions as to why the transfers are slower (even though that will likely cause my post to be deleted):
Judging by the output of
rclone --dump bodies it seems that
MLSD command after each transferred file. That appears to take considerable amount of time since I'm syncing directories having 4k-8k files. For most files data transfer takes a fraction of a time needed to do directory listing. I suppose
MLSD is issued to confirm file was transferred correctly. If that's the case, would it be possible to defer this validation to the end of whole directory sync? So suppose 100 files need to be synced in single directory:
MLSDcommand is sent to figure 100 files that are different
- 100 files are transferred
MLSDcommand is issued to confirm if files match
That would possibly save 99 directory listings.