Ftp: empty directory yields "directory not found" error

What is the problem you are having with rclone?

When doing a ls operation like rclone ls my_ftp:/, an error is logged:

2019/09/27 14:05:47 ERROR : : error listing: directory not found
2019/09/27 14:05:47 Failed to ls with 2 errors: last error was: directory not found

This happens only when the directory is empty. I did a tcpdump and found that rclone is doing a LIST / and receives an empty file list (hidden files are not included). So the error message is rather deceiving.

What is your rclone version (output from rclone version)

rclone v1.49.3-heads/v1.49.3-beta
- os/arch: linux/amd64
- go version: go1.13

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Official Docker image

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone ls my_ftp:/

A log from the command with the -vv flag (eg output from rclone -vv copy /tmp remote:tmp)

2019/09/27 14:18:39 DEBUG : rclone: Version "v1.49.3-heads/v1.49.3-beta" starting with parameters ["./rclone" "-vv" "--config" "./importer/rclone.conf" "ls" "my_ftp:/"]
2019/09/27 14:18:39 DEBUG : Using config file from "/root/importer/rclone.conf"
2019/09/27 14:18:39 DEBUG : ftp://x.x.x.x:21: Connecting to FTP server
2019/09/27 14:18:39 ERROR : : error listing: directory not found
2019/09/27 14:18:39 Failed to ls with 2 errors: last error was: directory not found

Which FTP server is this? Can you give instructions on how to replicate?

Hi! So I experienced this with several different ftp servers but vsftpd suffices for replication:

docker run --rm -d --name vsftpd -e "FTP_PASS=unsafe" fauria/vsftpd
export FTP_HOST=`docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' vsftpd`

echo "[some_ftp]\ntype = ftp" > rclone.conf
docker run --rm -v $(pwd)/rclone.conf:/root/.config/rclone/rclone.conf \
           rclone/rclone \
           --ftp-host $FTP_HOST \
           --ftp-user admin \
           --ftp-pass CetaJnfWmiH3ygmDwIFz2a25r1kUZA \
           ls some_ftp:/

And here's the corresponding tcpflow trace where it's apparent that the directory exists but is empty:

172.017.000.002.00021-172.017.000.003.59974: 220 (vsFTPd 3.0.2)
172.017.000.003.59974-172.017.000.002.00021: FEAT
172.017.000.002.00021-172.017.000.003.59974: 211-Features:
172.017.000.002.00021-172.017.000.003.59974:  EPRT
172.017.000.002.00021-172.017.000.003.59974:  EPSV
172.017.000.002.00021-172.017.000.003.59974:  MDTM
172.017.000.002.00021-172.017.000.003.59974:  PASV
172.017.000.002.00021-172.017.000.003.59974:  REST STREAM
172.017.000.002.00021-172.017.000.003.59974:  SIZE
172.017.000.002.00021-172.017.000.003.59974:  TVFS
172.017.000.002.00021-172.017.000.003.59974:  UTF8
172.017.000.002.00021-172.017.000.003.59974: 211 End
172.017.000.003.59974-172.017.000.002.00021: USER admin
172.017.000.002.00021-172.017.000.003.59974: 331 Please specify the password.
172.017.000.003.59974-172.017.000.002.00021: PASS unsafe
172.017.000.002.00021-172.017.000.003.59974: 230 Login successful.
172.017.000.003.59974-172.017.000.002.00021: TYPE I
172.017.000.002.00021-172.017.000.003.59974: 200 Switching to Binary mode.
172.017.000.003.59974-172.017.000.002.00021: OPTS UTF8 ON
172.017.000.002.00021-172.017.000.003.59974: 200 Always in UTF8 mode.
172.017.000.003.59974-172.017.000.002.00021: EPSV
172.017.000.002.00021-172.017.000.003.59974: 229 Entering Extended Passive Mode (|||21104|).
172.017.000.003.59974-172.017.000.002.00021: LIST /
172.017.000.002.00021-172.017.000.003.59974: 150 Here comes the directory listing.
172.017.000.002.00021-172.017.000.003.59974: 226 Directory send OK.
172.017.000.003.59974-172.017.000.002.00021: EPSV
172.017.000.002.00021-172.017.000.003.59974: 229 Entering Extended Passive Mode (|||21101|).
172.017.000.003.59974-172.017.000.002.00021: LIST /
172.017.000.002.00021-172.017.000.003.59974: 150 Here comes the directory listing.
172.017.000.002.00021-172.017.000.003.59974: 226 Directory send OK.
172.017.000.003.59974-172.017.000.002.00021: EPSV
172.017.000.002.00021-172.017.000.003.59974: 229 Entering Extended Passive Mode (|||21101|).
172.017.000.003.59974-172.017.000.002.00021: LIST /
172.017.000.002.00021-172.017.000.003.59974: 150 Here comes the directory listing.
172.017.000.002.00021-172.017.000.003.59974: 226 Directory send OK.

Thanks for your replication - I can reproduce that here.

A great idea running the FTP server in a docker container - I should probably do that in the integration test server, would make things a lot easier!

Note that you can drop the "echo... > rclone.conf" and replace some_ftp: with :ftp: to use an on the fly configured backend.

tcpflow is a great tool I've not come across before - I usually use ethereal for this which is much less convenient.

Back to the problem...

The fundamental problem is that the FTP protocol makes it impossible to tell an non existent directory from an empty directory (this is with vsftp)

ftp> ls empty
200 PORT command successful
150 Connecting to port 55253
226-Options: -l 
226 0 matches total
ftp> ls notfound
200 PORT command successful
150 Connecting to port 36867
226-Options: -l 
226 0 matches total
ftp> 

So what rclone does is when it does a listing and receives 0 entries back it tries to work out whether the source exists or not.

It does this by listing the parent and seeing whether the entry exists in the parent and this works fine as you'll see if you create an empty subdirectory.

However if the root is empty (as in this case) then rclone has no way of listing the parent and it is doing the wrong thing at the moment.

This is easy to fix though - can you have a go with this?

https://beta.rclone.org/branch/v1.49.3-101-g917cb2f5-fix-ftp-root-beta/ (uploaded in 15-30 mins)

Thanks for taking the time to debug and fix this!

Unfortunately, I can still reproduce the behaviour with that release. Is it possible that fullPath is actually "/" in such a case (instead of "")? I'm just guessing, though^^

Are you using a leading / in your remote names so remote:/path - if so then it will be / and that would explain it....

Try this which looks out for / too.

https://beta.rclone.org/branch/v1.49.3-101-gfef8b98b-fix-ftp-root-beta/ (uploaded in 15-30 mins)

Perfect, that release works. Thank you!

Thanks for testing :slight_smile: I'll merge that into the latest beta now - it will be there in 15-30 mins. Docker hub should build a rclone/rclone:beta from it too.

1 Like

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.