rclone copy FTP to S3 resulting in "No such file or directory" message from FTP server. It appears that the FTP server includes the entire path when you ls into a directory, so rclone is duplicating the path twice when attempting to download the file. download/batch/download\batch\146233701.K531L7N7.835
Here's a sample output from rclone ls tmhp:/download/batch on the FTP server:
[wasabi]
type = s3
provider = Wasabi
env_auth = false
access_key_id = **
secret_access_key = **
region =
endpoint = s3.us-central-1.wasabisys.com
location_constraint =
acl =
server_side_encryption =
storage_class =
[tmhp]
type = ftp
host = edi.tmhp.com
user = **
port = 21
pass = **
A log from the command with the -vv flag
2021/10/12 17:00:12 DEBUG : rclone: Version "v1.56.2" starting with parameters ["rclone" "copy" "tmhp:download/batch" "wasabi:apc-edi-files" "-vv"]
2021/10/12 17:00:12 DEBUG : Creating backend with remote "tmhp:download/batch"
2021/10/12 17:00:12 DEBUG : Using config file from "/home/sysacct/.config/rclone/rclone.conf"
2021/10/12 17:00:12 DEBUG : ftp://edi.tmhp.com:21/download/batch: Connecting to FTP server
2021/10/12 17:00:12 DEBUG : Creating backend with remote "wasabi:apc-edi-files"
2021/10/12 17:00:13 DEBUG : ftp://edi.tmhp.com:21/download/batch: Connecting to FTP server
2021/10/12 17:00:13 DEBUG : ftp://edi.tmhp.com:21/download/batch: Connecting to FTP server
2021/10/12 17:00:13 DEBUG : ftp://edi.tmhp.com:21/download/batch: Connecting to FTP server
2021/10/12 17:00:13 DEBUG : S3 bucket apc-edi-files: Waiting for checks to finish
2021/10/12 17:00:13 DEBUG : S3 bucket apc-edi-files: Waiting for transfers to finish
2021/10/12 17:00:13 ERROR : download\batch\146233701.K531L7N7.835: Failed to copy: failed to open source object: open: 550 download/batch/download\batch\146233701.K531L7N7.835: Cannot open file vexists: No such file or directory
2021/10/12 17:00:13 ERROR : download\batch\146233701.K56115QM.999: Failed to copy: failed to open source object: open: 550 download/batch/download\batch\146233701.K56115QM.999: Cannot open file vexists: No such file or directory
I think that is probably the problem - FTP servers should always work with / paths (Unix style)... (Though there are lots of not very standards compliant FTP servers out there!)
This FTP server belongs to the State of Texas, so I'm not sure what type it is.
As you mention, due to the backslashes, I am also convinced it's on Windows.
This FTP server works fine from any GUI client I have used, FileZilla, CyberDuck, etc.
It also works from the Linux ftp utility. The ftp ls command also returns just the file names, it does not return the full path with \ as it does with rclone ls.
Examples...
Linux ftp ls:
ftp> ls /download/batch
200 PORT command successful.
150 Opening ASCII connection for /download/batch
-rwxrwxrwx 1 owner group 6200 Sep 16 10:56 146233701.K531L7N7.835
-rwxrwxrwx 1 owner group 0 Sep 13 10:34 146233701.K56115QM.210913093817362_570317.dat.BID
-rwxrwxrwx 1 owner group 945 Sep 13 11:02 146233701.K56115QM.277CA
rclone ls:
$ rclone ls tmhp:/download/batch
6200 download\batch\146233701.K531L7N7.835
0 download\batch\146233701.K56115QM.210913093817362_570317.dat.BID
945 download\batch\146233701.K56115QM.277CA
The rclone output keeps the entire path in the listing, whereas other FTP utilities I try successfully navigate their way around the backslashes.
Having build cross-platform software for many years, I have found myself having to accommodate this very issue between *nix-style platforms and Windows to make everything work.
Might I suggest rclone get updated to work well with both forward and backward slashes?
Well, it is using plain text FTP, but over a VPN connection, so it's actually secure, but definitely inconvenient. The Medical field still heavily relies on the EDI file transfers, so I'm stuck in this world!
About WSL, that's an interesting thought. I will try it on a straight Linux box, maybe that will be different.
I'll try your other suggestions... maybe we can get this to work after all and I don't have to build my own sync app.
// FromStandardPath takes a / separated path in Standard encoding
// and converts it to a / separated path in the given encoding.
func FromStandardPath(e Encoder, s string) string {
if e == Standard {
return s
}
parts := strings.Split(s, "/")
encoded := make([]string, len(parts))
changed := false
for i, p := range parts {
enc := FromStandardName(e, p)
changed = changed || enc != p
encoded[i] = enc
}
if !changed {
return s
}
return strings.Join(encoded, "/")
}
In my experience/opinion hard-coding the "/" is not flexible enough to handle the varied FTP servers that exist on Windows... which I would guess to be in the millions (and shrinking). Maybe I'm wrong.
I am not proficient in Go or I would fix it in a PR, but I'll at least put in an issue to see where that goes.
In the meantime, I guess I need to find or build another solution.
That function is a bit too deep; at that point paths should be in internal rclone format - i.e. always / separated.
I think the fix will have to be somewhere within the ftp implementation. Wild guess: Adding a call to filepath.ToSlash() around entry.Name and entry.Target here:
Do you have build environment, so you can try out such a change in source yourself?
Yes, I found that section one call up from func FromStandardPath. I agree that would seem to be the best spot to address the issue.
I do not have a build environment setup for Go, nor have I ever coded in Go. I imagine I'd pick it up quickly, but I am pretty sure I can build an FTP to S3 copy utility that fits our business need in C# quicker, simply because I can hit the ground running without worrying about a new language / environment.
