Unclosed rclone FTP client connections swamp server

Suspected Bug
1

What is the problem you are having with rclone?

rclone doesn't seem to quit the FTP session and therefore close the connection to an FTP server. After many connections (we sync a folder every 5 minutes), the server cannot accept more incoming connections and blocks any clients.

What is your rclone version (output from rclone version)

rclone v1.50.2

  • os/arch: linux/amd64
  • go version: go1.13.4

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Amazon Linux 2 , Kernel 4.14.146-120.181.amzn2.x86_64

Which cloud storage system are you using? (eg Google Drive)

FTP with a very simple config:
[remote-ftp]
type = ftp
host = HOSTNAME
user = ****
pass = ****

The command you were trying to run (eg rclone copy /tmp remote:tmp)

Every 5 minutes we run:
rclone -v --update sync LOCALPATH remote-ftp:FOLDER/

A log from the command with the -vv flag (eg output from rclone -vv copy /tmp remote:tmp)

2020/01/15 12:55:30 DEBUG : rclone: Version "v1.50.2" starting with parameters ["rclone" "-vv" "--update" "sync" "LOCALPATH" "remote-ftp:FOLDER/"]
2020/01/15 12:55:30 DEBUG : Using config file from "/home/ec2-user/.config/rclone/rclone.conf"
2020/01/15 12:55:30 DEBUG : ftp://HOSTNAME:21/FOLDER: Connecting to FTP server
2020/01/15 12:55:34 DEBUG : 20191220/foo.jpg: Destination is newer than source, skipping
2020/01/15 12:55:34 DEBUG : 20191220/bar.jpg: Destination is newer than source, skipping
...
2020/01/15 12:55:34 INFO  : ftp://HOSTNAME:21/FOLDER: Waiting for checks to finish
2020/01/15 12:55:34 INFO  : ftp://HOSTNAME:21/FOLDER: Waiting for transfers to finish
2020/01/15 12:55:34 INFO  : Waiting for deletions to finish
2020/01/15 12:55:34 INFO  : 
Transferred:   	         0 / 0 Bytes, -, 0 Bytes/s, ETA -
Errors:                 0
Checks:                30 / 30, 100%
Transferred:            0 / 0, -
Elapsed time:          0s

2020/01/15 12:55:34 DEBUG : 4 go routines active
2020/01/15 12:55:34 DEBUG : rclone: Version "v1.50.2" finishing with parameters ["rclone" "-vv" "--update" "sync" "LOCALPATH" "remote-ftp:FOLDER/"]

Additional comparison of FTP commands sent by 'rclone' and the 'ftp' command line tool

Steps:

  1. listen to FTP traffic with tcpdump
  2. run 'rclone lsd remote-ftp:'
  3. run 'ftp HOSTNAME', log in, issue the command 'ls', and exit with 'quit'

Communication by rclone:
FTP: 220 (vsFTPd 3.0.3)
FTP: FEAT
FTP: 211-Features:
FTP: EPRT
FTP: EPSV
FTP: MDTM
FTP: PASV
FTP: REST STREAM
FTP: SIZE
FTP: TVFS
FTP: 211 End
FTP: USER ****
FTP: 331 Please specify the password.
FTP: PASS ****
FTP: 230 Login successful.
FTP: TYPE I
FTP: 200 Switching to Binary mode.
FTP: EPSV
FTP: 229 Entering Extended Passive Mode (|||63422|)
FTP: LIST
FTP: 150 Here comes the directory listing.
FTP: 226 Directory send OK.

Communication by ftp:
FTP: 220 (vsFTPd 3.0.3)
FTP: USER ****
FTP: 331 Please specify the password.
FTP: PASS ****
FTP: 230 Login successful.
FTP: SYST
FTP: 215 UNIX Type: L8
FTP: PASV
FTP: 227 Entering Passive Mode (91,220,179,245,124,68).
FTP: LIST
FTP: 150 Here comes the directory listing.
FTP: 226 Directory send OK.
FTP: QUIT
FTP: 221 Goodbye.

Obviously, the QUIT is part of the communication by 'ftp', because it was issued manually. However, it is missing from rclone's communication and may be the reason why we are experiencing this problem. Please mind: both in this test scenario, and when calling rclone every 5 minutes, the rclone call returns. So, it doesn't block, but somehow seems to omit the QUIT, maybe because it wants to do kind of a 'keep alive'?

Any hints much appreciated. If this qualifies as an actual bug, I'm happy to report it on GitHub.

Thanks
Florian

2

As a work around, you can use:

https://rclone.org/ftp/#ftp-concurrency

I think that might be an item for the module that is used in rclone:

But I am not sure offhand as I can let @ncw confirm.

3

When the rclone process quits the OS should close the TCP connection. The FTP server should notice this and release the resources it is using.

From the rfc

        An unexpected close on the control connection will cause the
        server to take the effective action of an abort (ABOR) and a
        logout (QUIT).

So I think rclone is behaving in a conformant way. You are right, rclone keeps the connections open in case it needs them again and relies on the OS closing the TCP connections when it quits. It doesn't send the QUIT command.

After the rclone process has quit can you check netstat -tuanp to see if the socket has closed properly (I can't think why it shouldn't have done).

Which FTP server are you using? If you have access to it you could look at netstat -tuanp or equivalent and see whether it things the rclone socket has been closed. If it doesn't then there may be a firewalling problem between the two servers.

rclone could probably be modified to QUIT the channels on exit, however I don't think it should need to be. Any FTP server which doesn't discard connections after a TCP close can easily be the victim of a denial-of-service attack.

4

Thank you for the detailed reply. I understand the possible causes and understand your position towards implementing this in rclone. We'll have a closer look into the conditions on the server side. Thanks for the helpful hints regarding the firewall and netstat.

5

Let me know how you get on, I'd like to know what you find.

6

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.