One general thought before I get to my problem: rclone must be the most discouraging project that I've ever encountered when it comes to raising issues. I have literally forced myself to create this post after giving up on GitHub
What is the problem you are having with rclone?
I'm trying to switch my file sync to rclone
with FTP backend (FastMail) but it seems to be extremely slow. FastMail provides WebDav backend as well which is at least 5-6x faster than FTP however lack of modification time synchronization in WebDav is kind of a blocker.
What is your rclone version (output from rclone version
)
$ rclone version
rclone 1.57.0
- os/version: pld 3.0
- os/kernel: 5.15.6-1 (armv7l)
- os/type: linux
- os/arch: arm
- go/version: go1.16.8
- go/linking: dynamic
- go/tags: none
Which cloud storage system are you using? (eg Google Drive)
FTP
The command you were trying to run (eg rclone copy /tmp remote:tmp
)
rclone --transfers 2 -P /dir remote:/dir
The rclone config contents with secrets removed.
[remote]
type = ftp
host = ftp.fastmail.com
user = user
pass = pass
disable_epsv = true
Instead of rclone -vv
let me share my suspicions as to why the transfers are slower (even though that will likely cause my post to be deleted):
Judging by the output of rclone --dump bodies
it seems that rclone
sends MLSD
command after each transferred file. That appears to take considerable amount of time since I'm syncing directories having 4k-8k files. For most files data transfer takes a fraction of a time needed to do directory listing. I suppose MLSD
is issued to confirm file was transferred correctly. If that's the case, would it be possible to defer this validation to the end of whole directory sync? So suppose 100 files need to be synced in single directory:
- initial
MLSD
command is sent to figure 100 files that are different - 100 files are transferred
- final
MLSD
command is issued to confirm if files match
That would possibly save 99 directory listings.