FTP No such file or directory

What is the problem you are having with rclone?

rclone copy FTP to S3 resulting in "No such file or directory" message from FTP server. It appears that the FTP server includes the entire path when you ls into a directory, so rclone is duplicating the path twice when attempting to download the file. download/batch/download\batch\146233701.K531L7N7.835

Here's a sample output from rclone ls tmhp:/download/batch on the FTP server:

     6200 download\batch\146233701.K531L7N7.835
    57538 download\batch\146233701.K551056P.277CAU
        0 download\batch\146233701.K56115QM.210913093817362_570317.dat.BID
      945 download\batch\146233701.K56115QM.277CA
      298 download\batch\146233701.K56115QM.999
        0 download\batch\146233701.K5615JUC.210913104901413_4026910.dat.BID

I would expect it to only list the file names without the entire path prefix.

Is there a way to handle this weirdness in rclone?

What is your rclone version (output from rclone version)

rclone v1.56.2
- os/version: ubuntu 20.04 (64 bit)
- os/kernel: 5.4.72-microsoft-standard-WSL2 (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.16.8
- go/linking: static
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

FTP and Wasabi S3

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copy tmhp:download/batch wasabi:apc-edi-files

The rclone config contents with secrets removed.

[wasabi]
type = s3
provider = Wasabi
env_auth = false
access_key_id = **
secret_access_key = **
region =
endpoint = s3.us-central-1.wasabisys.com
location_constraint =
acl =
server_side_encryption =
storage_class =

[tmhp]
type = ftp
host = edi.tmhp.com
user = **
port = 21
pass = **

A log from the command with the -vv flag

2021/10/12 17:00:12 DEBUG : rclone: Version "v1.56.2" starting with parameters ["rclone" "copy" "tmhp:download/batch" "wasabi:apc-edi-files" "-vv"]
2021/10/12 17:00:12 DEBUG : Creating backend with remote "tmhp:download/batch"
2021/10/12 17:00:12 DEBUG : Using config file from "/home/sysacct/.config/rclone/rclone.conf"
2021/10/12 17:00:12 DEBUG : ftp://edi.tmhp.com:21/download/batch: Connecting to FTP server
2021/10/12 17:00:12 DEBUG : Creating backend with remote "wasabi:apc-edi-files"
2021/10/12 17:00:13 DEBUG : ftp://edi.tmhp.com:21/download/batch: Connecting to FTP server
2021/10/12 17:00:13 DEBUG : ftp://edi.tmhp.com:21/download/batch: Connecting to FTP server
2021/10/12 17:00:13 DEBUG : ftp://edi.tmhp.com:21/download/batch: Connecting to FTP server
2021/10/12 17:00:13 DEBUG : S3 bucket apc-edi-files: Waiting for checks to finish
2021/10/12 17:00:13 DEBUG : S3 bucket apc-edi-files: Waiting for transfers to finish
2021/10/12 17:00:13 ERROR : download\batch\146233701.K531L7N7.835: Failed to copy: failed to open source object: open: 550 download/batch/download\batch\146233701.K531L7N7.835: Cannot open file vexists: No such file or directory
2021/10/12 17:00:13 ERROR : download\batch\146233701.K56115QM.999: Failed to copy: failed to open source object: open: 550 download/batch/download\batch\146233701.K56115QM.999: Cannot open file vexists: No such file or directory

hello and welcome to the forum,

that ftp server, is it proftpd, pureftpd, cerberus or what?
what is the operating system that hosts that ftp server?

are you able to access the ftp server using a ftp client such as filezilla?

that path contains / and \ - is that intentional?

the path of the command you posted does not match the path of the command in the debug log
tmhp:/download/batch
and
tmhp:download/batch

what is the output of
rclone lsd tmhp:
rclone lsd tmhp:/

I think that is probably the problem - FTP servers should always work with / paths (Unix style)... (Though there are lots of not very standards compliant FTP servers out there!)

Do you know which FTP server this is?

i asked the OP, so far no answer, but based on the paths, i am guesstimating it is windows server.

I appreciate the responses.

This FTP server belongs to the State of Texas, so I'm not sure what type it is.

As you mention, due to the backslashes, I am also convinced it's on Windows.

This FTP server works fine from any GUI client I have used, FileZilla, CyberDuck, etc.

It also works from the Linux ftp utility. The ftp ls command also returns just the file names, it does not return the full path with \ as it does with rclone ls.

Examples...

Linux ftp ls:

ftp> ls /download/batch
200 PORT command successful.
150 Opening ASCII connection for /download/batch
-rwxrwxrwx   1 owner    group            6200 Sep 16 10:56 146233701.K531L7N7.835
-rwxrwxrwx   1 owner    group               0 Sep 13 10:34 146233701.K56115QM.210913093817362_570317.dat.BID
-rwxrwxrwx   1 owner    group             945 Sep 13 11:02 146233701.K56115QM.277CA

rclone ls:

$ rclone ls tmhp:/download/batch
     6200 download\batch\146233701.K531L7N7.835
        0 download\batch\146233701.K56115QM.210913093817362_570317.dat.BID
      945 download\batch\146233701.K56115QM.277CA

The rclone output keeps the entire path in the listing, whereas other FTP utilities I try successfully navigate their way around the backslashes.

Having build cross-platform software for many years, I have found myself having to accommodate this very issue between *nix-style platforms and Windows to make everything work.

Might I suggest rclone get updated to work well with both forward and backward slashes?

yeah, it makes sense,
if the state of texas transfers medicaid data over insecure, not reliable ftp, sure, let's make it worse, by using windowz :wink:

when you connect using filezilla, do you see something like
Response: fzSftp started, protocol_version=11

there are some flags that might give more detailed info
https://rclone.org/ftp/#limitations
--dump headers, --dump bodies, --dump auth for debugging

i wonder if there is an issue using wsl?
have you tried rclone on the windows os?

Well, it is using plain text FTP, but over a VPN connection, so it's actually secure, but definitely inconvenient. The Medical field still heavily relies on the EDI file transfers, so I'm stuck in this world!

About WSL, that's an interesting thought. I will try it on a straight Linux box, maybe that will be different.

I'll try your other suggestions... maybe we can get this to work after all and I don't have to build my own sync app.

Will report results.

Thanks

Unfortunately, same results from a non-WSL Linux machine.

I found the code that is causing this:

https://github.com/rclone/rclone/blob/cf9b82b8dbfe0b4ae8b89059a1eb10d97c35881a/lib/encoder/encoder.go#L1123

// FromStandardPath takes a / separated path in Standard encoding
// and converts it to a / separated path in the given encoding.
func FromStandardPath(e Encoder, s string) string {
	if e == Standard {
		return s
	}
	parts := strings.Split(s, "/")
	encoded := make([]string, len(parts))
	changed := false
	for i, p := range parts {
		enc := FromStandardName(e, p)
		changed = changed || enc != p
		encoded[i] = enc
	}
	if !changed {
		return s
	}
	return strings.Join(encoded, "/")
}

In my experience/opinion hard-coding the "/" is not flexible enough to handle the varied FTP servers that exist on Windows... which I would guess to be in the millions (and shrinking). Maybe I'm wrong.

I am not proficient in Go or I would fix it in a PR, but I'll at least put in an issue to see where that goes.

In the meantime, I guess I need to find or build another solution.

Thanks for the help guys.

That function is a bit too deep; at that point paths should be in internal rclone format - i.e. always / separated.

I think the fix will have to be somewhere within the ftp implementation. Wild guess: Adding a call to filepath.ToSlash() around entry.Name and entry.Target here:

Do you have build environment, so you can try out such a change in source yourself?

Yes, I found that section one call up from func FromStandardPath. I agree that would seem to be the best spot to address the issue.

I do not have a build environment setup for Go, nor have I ever coded in Go. I imagine I'd pick it up quickly, but I am pretty sure I can build an FTP to S3 copy utility that fits our business need in C# quicker, simply because I can hit the ground running without worrying about a new language / environment.

Thanks for the suggestion, though.