FTP sync problem big files (re-transfer)

Help and Support
1

Hi community,

I have spent a lot of time testing and trying different stuff to get my setup to work and I am hoping someone can point me in the right direction to solve my issue:

running sync with crypt from LXC (debian) over local LAN to vsftpd FTP server (plain, unencrypted) on Pi4 (ubuntu)
everything runs fine except for big files (9-10GB up) get transferred completely and then immediately starts over again.
exactly like FTP upload and download for big files (over 10GB) fails · Issue #5328 · rclone/rclone · GitHub

disabling EPSV as in above issue did not resolve my problem.
I also tried disabling MLSD.
sometimes (I cannot figure out when/how) when tested it works as intended, but mostly not. FTPing with filezilla to server works fine.

for testing both machines with no firewall
Vers: 1.55.1

rclone sync /mnt/crownstoner/$FOLDER/ rbackup:/$FOLDER/ --track-renames --track-renames-strategy leaf --backup-dir rbackup:/.recycle_bin --log-level DEBUG --log-file ./$LOGNAME.log

[rbackupftp]
type = ftp
host = 192.168.8.68
user = upload
pass = *** ENCRYPTED ***
tls = false
explicit_tls = false
no_check_certificate = true
disable_epsv = true

2021/05/26 16:18:31 DEBUG : Musikprojekte.tar.bin: Removed after failed upload: read tcp 192.168.8.110:56658->192.168.8.68:21: i/o timeout
2021/05/26 16:18:31 DEBUG : Musikprojekte.tar: Received error: update stor: read tcp 192.168.8.110:56658->192.168.8.68:21: i/o timeout - low level retry 3/10

Cheers, T

2

can you post the full debug log that inlcudes the error
the name of the remote in the command does not match the name of the remote in the config file.

3

sorry, for the confusion: the config I showed is the config for the ftp remote. the one in the command is my crypt remote that points to that ftp remote.

This here is the complete log file where the transfer worked:
https://pastebin.com/NwiAkPSJ

with the exact same command (with a few more files) this happens (until the point where I cancel it, because otherwise it would run for days):
https://pastebin.com/d5isw4kv

4

sorry, i gave up on unreliable ftp many years ago, so i cannot help much with that.

not sure you know that rclone can act as a ftp server.
better yet rclone can act as sftp server, which can use checksums to ensure files are transferred without corruption and crypt the data during transit.

5

thanks for trying though.
its already planned to run on peer to peer VPN so any additional layer of encryption just kills performance (end point is a Pi4).
Maybe you have an alternative suggestion what I can use as "lightweight endpoint" over not so great internet? The aim is to have my Pi with an external hard drive as off-site backup location. It will run within a VPN connection.

6

try rclone serve sftp remote: or any sftp server, that could checksum the data as ftp cannot be trusted for backups.

1 Like
7

It would be worth giving the latest beta a try as that has fixes to the FTP backend.

1 Like
8

@tomstull I had exactly the same issue as you and almost the same setup (rclone @ raspbian <-> vsftpd @ openwrt). Following @ncw's advice and using latest beta solved it.

pi@ZEROPI:~ $ rclone version
rclone v1.56.0-beta.5554.e635f4c0b
- os/version: raspbian 10.9
- os/kernel: 5.10.17+ (armv6l)
- os/type: linux
- os/arch: arm
- go/version: go1.16.5
- go/linking: static
- go/tags: none

remote setup:

[ibox-ftp]
type = ftp
host = 192.xxx.xxx.xxx
user = xxx
pass = *** ENCRYPTED ***
port = 21
disable_epsv = true
concurrency = 1
no_check_certificate = true
disable_mlsd = true
idle_timeout = 5s
close_timeout = 5s

That setup is a bit slower; ~10% maybe but at least it works...

1 Like
9

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.