Rclone sync with ftp serve on the remote host

Huy_Banh · May 18, 2023, 7:59pm

What is the problem you are having with rclone?

I am doing "rclone sync" with a remote served by "rclone serve ftp". It shows errors as if it is not connecting to the specified port 9005, but some random ports:

It shows errors as if it is not connecting to the specified port 9005, but some random ports.

Run the command 'rclone version' and share the full output of the command.

rclone version
rclone v1.62.2

os/version: userful 8 (64 bit)
os/kernel: 4.18.0-448.el8.userful.x86_64 (x86_64)
os/type: linux
os/arch: amd64
go/version: go1.20.2
go/linking: static
go/tags: none

Which cloud storage system are you using? (eg Google Drive)

FTP, with clone ftp serve:
sudo rclone serve ftp --addr :9005 --user ha --pass 1234 /var/source-content/

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

rclone sync --interactive /var/source-content/ ha_peer:/

The rclone config contents with secrets removed.

Editing existing "ha_peer" remote with options:
- type: ftp
- host: 192.168.163.99
- user: ha
- port: 9005
- pass: *** ENCRYPTED ***

A log from the command with the `-vv` flag

2023/05/18 12:56:39 DEBUG : rclone: Version "v1.62.2" starting with parameters ["rclone" "sync" "-vv" "/var/source-content/" "ha_peer:/"]
2023/05/18 12:56:39 DEBUG : Creating backend with remote "/var/source-content/"
2023/05/18 12:56:39 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2023/05/18 12:56:39 DEBUG : fs cache: renaming cache item "/var/source-content/" to be canonical "/var/source-content"
2023/05/18 12:56:39 DEBUG : Creating backend with remote "ha_peer:/"
2023/05/18 12:56:39 DEBUG : ftp://192.168.163.99:9005: Connecting to FTP server
2023/05/18 12:56:39 DEBUG : ftp://192.168.163.99:9005: dial("tcp","192.168.163.99:9005")
2023/05/18 12:56:39 DEBUG : ftp://192.168.163.99:9005: > dial: conn=*fshttp.timeoutConn, err=<nil>
2023/05/18 12:56:39 DEBUG : ftp://192.168.163.99:9005: dial("tcp","192.168.163.99:30511")
2023/05/18 12:56:39 DEBUG : ftp://192.168.163.99:9005: > dial: conn=<nil>, err=dial tcp 192.168.163.99:30511: connect: no route to host
2023/05/18 12:56:39 ERROR : ftp://192.168.163.99:9005: error reading destination root directory: dial tcp 192.168.163.99:30511: connect: no route to host
2023/05/18 12:56:39 DEBUG : ftp://192.168.163.99:9005: Waiting for checks to finish
2023/05/18 12:56:39 DEBUG : ftp://192.168.163.99:9005: Waiting for transfers to finish
2023/05/18 12:56:39 ERROR : ftp://192.168.163.99:9005: not deleting files as there were IO errors
2023/05/18 12:56:39 ERROR : ftp://192.168.163.99:9005: not deleting directories as there were IO errors
2023/05/18 12:56:39 ERROR : Attempt 1/3 failed with 1 errors and: dial tcp 192.168.163.99:30511: connect: no route to host
2023/05/18 12:56:39 DEBUG : ftp://192.168.163.99:9005: dial("tcp","192.168.163.99:30715")
2023/05/18 12:56:39 DEBUG : ftp://192.168.163.99:9005: > dial: conn=<nil>, err=dial tcp 192.168.163.99:30715: connect: no route to host
2023/05/18 12:56:39 ERROR : ftp://192.168.163.99:9005: error reading destination root directory: dial tcp 192.168.163.99:30715: connect: no route to host
2023/05/18 12:56:39 DEBUG : ftp://192.168.163.99:9005: Waiting for checks to finish
2023/05/18 12:56:39 DEBUG : ftp://192.168.163.99:9005: Waiting for transfers to finish
2023/05/18 12:56:39 ERROR : ftp://192.168.163.99:9005: not deleting files as there were IO errors
2023/05/18 12:56:39 ERROR : ftp://192.168.163.99:9005: not deleting directories as there were IO errors
2023/05/18 12:56:39 ERROR : Attempt 2/3 failed with 1 errors and: dial tcp 192.168.163.99:30715: connect: no route to host
2023/05/18 12:56:39 DEBUG : ftp://192.168.163.99:9005: dial("tcp","192.168.163.99:30974")
2023/05/18 12:56:39 DEBUG : ftp://192.168.163.99:9005: > dial: conn=<nil>, err=dial tcp 192.168.163.99:30974: connect: no route to host
2023/05/18 12:56:39 ERROR : ftp://192.168.163.99:9005: error reading destination root directory: dial tcp 192.168.163.99:30974: connect: no route to host
2023/05/18 12:56:39 DEBUG : ftp://192.168.163.99:9005: Waiting for checks to finish
2023/05/18 12:56:39 DEBUG : ftp://192.168.163.99:9005: Waiting for transfers to finish
2023/05/18 12:56:39 ERROR : ftp://192.168.163.99:9005: not deleting files as there were IO errors
2023/05/18 12:56:39 ERROR : ftp://192.168.163.99:9005: not deleting directories as there were IO errors
2023/05/18 12:56:39 ERROR : Attempt 3/3 failed with 1 errors and: dial tcp 192.168.163.99:30974: connect: no route to host
2023/05/18 12:56:39 INFO  : 
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Errors:                 1 (retrying may help)
Elapsed time:         0.0s

2023/05/18 12:56:39 DEBUG : 3 go routines active
2023/05/18 12:56:39 DEBUG : ftp://192.168.163.99:9005: closing 1 unused connections
2023/05/18 12:56:39 Failed to sync: dial tcp 192.168.163.99:30974: connect: no route to host

asdffdsa · May 18, 2023, 8:12pm

hello and welcome to the forum,

--- are the client and server on the same local subnet?
--- can filezilla or such tool access the ftp server?

Huy_Banh · May 18, 2023, 8:46pm

Thanks for your very quick response. They are on two different servers.

I changed the remote ftp to avoid random passive ports (it has firewall enabled):

sudo rclone serve ftp --addr :9005 --user ha --pass 1234 --passive-port 9006-9006 /var/source-content/

Still not working, and Filezilla also not working. Here is the log from Filezilla:

Status:	Connecting to 192.168.123.99:9005...
Status:	Connection established, waiting for welcome message...
Status:	Insecure server, it does not support FTP over TLS.
Status:	Logged in
Status:	Retrieving directory listing...
Command:	PWD
Response:	257 / is the current directory
Command:	TYPE I
Response:	200 Type set to binary
Command:	PASV
Error:	Connection timed out after 20 seconds of inactivity
Error:	Failed to retrieve directory listing
Status:	Disconnected from server

Any advice?

asdffdsa · May 18, 2023, 9:06pm

if the client and server both use rclone, try rclone serve sftp.
bonus:
--- file encryption during transfer.
--- checksum verification after transfer.

Huy_Banh · May 18, 2023, 9:14pm

Thank you for that.

kapitainsky · May 19, 2023, 6:40am

It is not random port by error but how ftp works - it always uses two ports - command one (in your case 9005) and data port (which you do not specify explicitly and rclone uses one from range 30000-32000):

It is very common problem people face when adding firewall to the ftp equation:)

https://www.cosmos.esa.int/documents/772136/977578/psa_activeVsPassiveFtp.pdf/5e36a7b8-8732-4e65-ab6b-6cf94a742ea6

Your Filezilla test shows exactly that it is the case. Command port works and server is instructed to send data but it is blocked by firewall and ftp client can not receive directory listing.

Long story short rclone has it covered, use in rclone serve:

# rclone serve ftp --help
...
--passive-port string                    Passive port range to use (default "30000-32000")

to fix second port and open it in your firewall in addition to 9005.

Also @asdffdsa suggestion is good one that sftp is better option (it also uses only one port). FTP is legacy protocol and unless you really need I would not use it. Of course choice is yours. rclone support both.

Huy_Banh · May 19, 2023, 4:13pm

Thank Kapitainsky for a very thorough explanation of FTP ports. I should have checked it first. Anyway, as Jojothehumanmoney and you suggested, I am moving to sftp and get rid of those ports and firewall issues.

Have a good day to you all.

system · May 22, 2023, 4:14pm

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.