Slow FTP sync with directories keeping lots of files

One general thought before I get to my problem: rclone must be the most discouraging project that I've ever encountered when it comes to raising issues. I have literally forced myself to create this post after giving up on GitHub :slight_smile:

What is the problem you are having with rclone?

I'm trying to switch my file sync to rclone with FTP backend (FastMail) but it seems to be extremely slow. FastMail provides WebDav backend as well which is at least 5-6x faster than FTP however lack of modification time synchronization in WebDav is kind of a blocker.

What is your rclone version (output from rclone version)

$ rclone version
rclone 1.57.0
- os/version: pld 3.0
- os/kernel: 5.15.6-1 (armv7l)
- os/type: linux
- os/arch: arm
- go/version: go1.16.8
- go/linking: dynamic
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

FTP

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone --transfers 2 -P /dir remote:/dir

The rclone config contents with secrets removed.

[remote]
type = ftp
host = ftp.fastmail.com
user = user
pass = pass
disable_epsv = true

Instead of rclone -vv let me share my suspicions as to why the transfers are slower (even though that will likely cause my post to be deleted):

Judging by the output of rclone --dump bodies it seems that rclone sends MLSD command after each transferred file. That appears to take considerable amount of time since I'm syncing directories having 4k-8k files. For most files data transfer takes a fraction of a time needed to do directory listing. I suppose MLSD is issued to confirm file was transferred correctly. If that's the case, would it be possible to defer this validation to the end of whole directory sync? So suppose 100 files need to be synced in single directory:

  • initial MLSD command is sent to figure 100 files that are different
  • 100 files are transferred
  • final MLSD command is issued to confirm if files match

That would possibly save 99 directory listings.

hello and welcome to the forum,

have you tested using

  • --ftp-disable-mlsd
  • --fast-list tho no idea if it works on ftp

have you tested using

  • --ftp-disable-mlsd

Yes but it's prohibitively slow -- check of first 1000 of files took over a minute so I skipped the other 7000 (that's for sync between equal directories so nothing has to be synchronized in practice)

  • --fast-list tho no idea if it works on ftp

It didn't seem to be implemented for FTP so I didn't try before posting, but tested now just to confirm and does not have any effect.

And to put it in perspective transfer are instantaneous while MLSD on directory with 8k files takes about 6 seconds. So for 100 files for which transfers take few seconds, 10 minutes is spent doing directory listings.

do not know your exact use-case but might try a work around, for example,

rclone check /dir remote:/dir --one-way --differ=differ.lst
rclone copy /dir remote:/dir --no-traverse --no-check-dest --files-from=differ.lst

Thanks but ultimately decided to stick to lftp which seems much more robust for this purpose.

ok, thanks for the update.

rclone can be slower than other tools, as rclone tries to validate the transfer.
for example, with s3, rclone will calculate the hash of the local file before upload.

Perhaps @ivandeex has an idea?

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.