Nodes with trailing whitespace fail to copy

laur · December 6, 2022, 12:20am

What is the problem you are having with rclone?

rclone copying nodes from FTP remote that have trailing whitespace fails.
Example:

$ ls
'dir-1-trailing '  ' dir-2-prefix-whitespace'

With the following command I'd expect both nodes to be copied, yet only the one with leading whitespace is.

Run the command 'rclone version' and share the full output of the command.

./rclone version
rclone v1.60.1
- os/version: debian bookworm/sid (64 bit)
- os/kernel: 6.0.0-4-amd64 (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.19.3
- go/linking: static
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

FTPS

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

rclone copy -L -vv --dump bodies seedbox:files/testdir

A log from the command with the `-vv` flag

$ ./rclone copy -L -vv --dump bodies seedbox:files/testdir  ./
<7>DEBUG : rclone: Version "v1.60.1" starting with parameters ["./rclone" "copy" "-L" "-vv" "--dump" "bodies" "seedbox:files/testdir" "./"]
<7>DEBUG : rclone: systemd logging support activated
<7>DEBUG : Creating backend with remote "seedbox:files/testdir"
<7>DEBUG : Using config file from "/home/laur89/.config/rclone/rclone.conf"
<7>DEBUG : ftp://remote.server:21/files/testdir: Connecting to FTP server
<7>DEBUG : ftp://remote.server:21/files/testdir: dial("tcp","remote.server:21")
<7>DEBUG : ftp://remote.server:21/files/testdir: > dial: conn=*fshttp.timeoutConn, err=<nil>
<7>DEBUG : FTP Rx: "220 (vsFTPd 3.0.3)"
<7>DEBUG : FTP Tx: "AUTH TLS"
<7>DEBUG : FTP Rx: "234 Proceed with negotiation."
<7>DEBUG : FTP Tx: "USER laur"
<7>DEBUG : FTP Rx: "331 Please specify the password."
<7>DEBUG : FTP Tx: PASS *****
<7>DEBUG : FTP Rx: "230 Login successful."
<7>DEBUG : FTP Tx: "FEAT"
<7>DEBUG : FTP Rx: "211-Features:"
<7>DEBUG : FTP Rx: " AUTH TLS"
<7>DEBUG : FTP Rx: " UTF8"
<7>DEBUG : FTP Rx: " EPRT"
<7>DEBUG : FTP Rx: " EPSV"
<7>DEBUG : FTP Rx: " MDTM"
<7>DEBUG : FTP Rx: " PASV"
<7>DEBUG : FTP Rx: " PBSZ"
<7>DEBUG : FTP Rx: " PROT"
<7>DEBUG : FTP Rx: " REST STREAM"
<7>DEBUG : FTP Rx: " SIZE"
<7>DEBUG : FTP Rx: " TVFS"
<7>DEBUG : FTP Rx: "211 End"
<7>DEBUG : FTP Tx: "TYPE I"
<7>DEBUG : FTP Rx: "200 Switching to Binary mode."
<7>DEBUG : FTP Tx: "OPTS UTF8 ON"
<7>DEBUG : FTP Rx: "200 Always in UTF8 mode."
<7>DEBUG : FTP Tx: "PBSZ 0"
<7>DEBUG : FTP Rx: "200 PBSZ set to 0."
<7>DEBUG : FTP Tx: "PROT P"
<7>DEBUG : FTP Rx: "200 PROT now Private."
<7>DEBUG : FTP Tx: "EPSV"
<7>DEBUG : FTP Rx: "229 Entering Extended Passive Mode (|||9258|)"
<7>DEBUG : ftp://remote.server:21/files/testdir: dial("tcp","8.8.8.80:9258")
<7>DEBUG : ftp://remote.server:21/files/testdir: > dial: conn=*tls.Conn, err=<nil>
<7>DEBUG : FTP Tx: "LIST files"
<7>DEBUG : FTP Rx: "150 Here comes the directory listing."
<7>DEBUG : FTP Rx: "drwxr-xr-x   13 1235     1235        16384 Dec 05 20:37 unrelated-dir-1"
<7>DEBUG : FTP Rx: "drwxr-xr-x    5 1235     1235         4096 Dec 06 00:47 unrelated-dir-2"
<7>DEBUG : FTP Rx: "drwxr-xr-x    2 1235     1235         4096 Oct 26 13:28 unrelated-dir-3"
<7>DEBUG : FTP Rx: "-rw-r--r--    1 1235     1235           14 Oct 25 18:13 some-file"
<7>DEBUG : FTP Rx: "drwxr-xr-x    3 1235     1235         4096 Dec 06 01:11 testdir"
<7>DEBUG : FTP Rx: "226 Directory send OK."
<7>DEBUG : Creating backend with remote "./"
<7>DEBUG : local: detected overridden config - adding "{12rtk}" suffix to name
<7>DEBUG : fs cache: renaming cache item "./" to be canonical "local{12rtk}:/tmp/rclone-test-dir"
<7>DEBUG : FTP Tx: "EPSV"
<7>DEBUG : FTP Rx: "229 Entering Extended Passive Mode (|||9518|)"
<7>DEBUG : ftp://remote.server:21/files/testdir: dial("tcp","8.8.8.80:9518")
<7>DEBUG : ftp://remote.server:21/files/testdir: > dial: conn=*tls.Conn, err=<nil>
<7>DEBUG : FTP Tx: "LIST files/testdir"
<7>DEBUG : FTP Rx: "150 Here comes the directory listing."
<7>DEBUG : FTP Rx: "-rw-r--r--    1 1235     1235            0 Dec 06 01:11  dir-2-prefix-whitespace"
<7>DEBUG : FTP Rx: "drwxr-xr-x    2 1235     1235         4096 Dec 06 01:09 dir-1-trailing "
<7>DEBUG : FTP Rx: "226 Directory send OK."
<7>DEBUG : FTP Tx: "EPSV"
<7>DEBUG : ftp://remote.server:21/files/testdir: Connecting to FTP server
<7>DEBUG : ftp://remote.server:21/files/testdir: dial("tcp","remote.server:21")
<7>DEBUG : FTP Rx: "229 Entering Extended Passive Mode (|||9853|)"
<7>DEBUG : ftp://remote.server:21/files/testdir: dial("tcp","8.8.8.80:9853")
<7>DEBUG : ftp://remote.server:21/files/testdir: > dial: conn=*fshttp.timeoutConn, err=<nil>
<7>DEBUG : ftp://remote.server:21/files/testdir: > dial: conn=*tls.Conn, err=<nil>
<7>DEBUG : FTP Tx: "LIST files/testdir/dir-1-trailing␠"
<7>DEBUG : FTP Rx: "220 (vsFTPd 3.0.3)"
<7>DEBUG : FTP Tx: "AUTH TLS"
<7>DEBUG : FTP Rx: "150 Here comes the directory listing."
<7>DEBUG : FTP Rx: "234 Proceed with negotiation."
<7>DEBUG : FTP Tx: "USER laur"
<7>DEBUG : FTP Rx: "331 Please specify the password."
<7>DEBUG : FTP Tx: PASS *****
<7>DEBUG : FTP Rx: "226 Directory send OK."
<7>DEBUG : FTP Tx: "EPSV"
<7>DEBUG : FTP Rx: "230 Login successful."
<7>DEBUG : FTP Tx: "FEAT"
<7>DEBUG : FTP Rx: "229 Entering Extended Passive Mode (|||6427|)"
<7>DEBUG : ftp://remote.server:21/files/testdir: dial("tcp","8.8.8.80:6427")
<7>DEBUG : FTP Rx: "211-Features:"
<7>DEBUG : FTP Rx: " AUTH TLS"
<7>DEBUG : FTP Rx: " UTF8"
<7>DEBUG : FTP Rx: " EPRT"
<7>DEBUG : FTP Rx: " EPSV"
<7>DEBUG : FTP Rx: " MDTM"
<7>DEBUG : FTP Rx: " PASV"
<7>DEBUG : FTP Rx: " PBSZ"
<7>DEBUG : FTP Rx: " PROT"
<7>DEBUG : FTP Rx: " REST STREAM"
<7>DEBUG : FTP Rx: " SIZE"
<7>DEBUG : FTP Rx: " TVFS"
<7>DEBUG : FTP Rx: "211 End"
<7>DEBUG : FTP Tx: "TYPE I"
<7>DEBUG : ftp://remote.server:21/files/testdir: > dial: conn=*tls.Conn, err=<nil>
<7>DEBUG : FTP Tx: "LIST files/testdir"
<7>DEBUG : FTP Rx: "200 Switching to Binary mode."
<7>DEBUG : FTP Tx: "OPTS UTF8 ON"
<7>DEBUG : FTP Rx: "150 Here comes the directory listing."
<7>DEBUG : FTP Rx: "200 Always in UTF8 mode."
<7>DEBUG : FTP Tx: "PBSZ 0"
<7>DEBUG : FTP Rx: "200 PBSZ set to 0."
<7>DEBUG : FTP Tx: "PROT P"
<7>DEBUG : FTP Rx: "200 PROT now Private."
<7>DEBUG : FTP Tx: "EPSV"
<7>DEBUG : FTP Rx: "-rw-r--r--    1 1235     1235            0 Dec 06 01:11  dir-2-prefix-whitespace"
<7>DEBUG : FTP Rx: "drwxr-xr-x    2 1235     1235         4096 Dec 06 01:09 dir-1-trailing "
<7>DEBUG : FTP Rx: "229 Entering Extended Passive Mode (|||7055|)"
<7>DEBUG : ftp://remote.server:21/files/testdir: dial("tcp","8.8.8.80:7055")
<7>DEBUG : ftp://remote.server:21/files/testdir: > dial: conn=*tls.Conn, err=<nil>
<7>DEBUG : FTP Tx: "RETR files/testdir/ dir-2-prefix-whitespace"
<7>DEBUG : FTP Rx: "226 Directory send OK."
<7>DEBUG : Local file system at /tmp/rclone-test-dir: Waiting for checks to finish
<7>DEBUG : Local file system at /tmp/rclone-test-dir: Waiting for transfers to finish
<7>DEBUG : FTP Rx: "150 Opening BINARY mode data connection for files/testdir/ dir-2-prefix-whitespace (0 bytes)."
<7>DEBUG : FTP Tx: "MDTM files/testdir/ dir-2-prefix-whitespace"
<7>DEBUG : FTP Rx: "213 20221206011119"
<7>DEBUG : FTP Rx: "226 Transfer complete."
<6>INFO  :  dir-2-prefix-whitespace: Copied (new)
<6>INFO  : 
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Transferred:            1 / 1, 100%
Elapsed time:         6.4s

<7>DEBUG : 3 go routines active
<7>DEBUG : ftp://remote.server:21/files/testdir: closing 2 unused connections
<7>DEBUG : FTP Tx: "QUIT"
<7>DEBUG : FTP Tx: "QUIT"

Is this expected behavior?

ncw · December 7, 2022, 2:54pm

FTP servers are really bad at this...

The default encoding for the FTP backend is

  --ftp-encoding MultiEncoder   The encoding for the backend (default Slash,Del,Ctl,RightSpace,Dot)

Note the RightSpace in there which tells rclone to encode trailing spaces as a unicode alternative.

You can try removing it and see if it works, so use

--ftp-encoding "Slash,Del,Ctl,Dot"

I suspect it won't but you might get lucky!

system · January 6, 2023, 2:55pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.