Sftp SSH_FX_FAILURE directory not found

Help and Support
1

looks like a similar issue to:

however, the links are broken and i cannot see the solution.

What is the problem you are having with rclone?

cannot copy to or from sftp server i have no control over.

What is your rclone version (output from rclone version)

rclone v1.52.2
- os/arch: linux/amd64
- go version: go1.14.4

Which OS you are using and how many bits (eg Windows 7, 64 bit)

ubuntu 18.04, 64bit

Which cloud storage system are you using? (eg Google Drive)

sftp

The command you were trying to run (eg rclone copy /tmp remote:tmp)

# to get the SSH_FX_FAILURE error
rclone copy test.txt eleadtrack:
# attempt to debug
rclone copyto  --low-level-retries 1 --retries 1 -vv eleadtrack:/Hendrick_SourceSubsource_USED_06252020.csv ./test.csv

The rclone config contents with secrets removed.

[eleadtrack]
type = sftp
host = example.com
user = <user>
pass = <encryptedpass>

A log from the command with the -vv flag

2020/06/25 07:50:58 DEBUG : rclone: Version "v1.52.2" starting with parameters ["rclone" "copyto" "--low-level-retries" "1" "--retries" "1" "-vv" "eleadtrack_old:/Hendrick_SourceSubsource_USED_06252020.csv" "./test.csv"]
2020/06/25 07:50:58 DEBUG : Using config file from "/home/ak_zuar01/.config/rclone/rclone.conf"
2020/06/25 07:50:58 DEBUG : sftp://user@example.com:22//Hendrick_SourceSubsource_USED_06252020.csv: New connection 1.2.3.4:43808->4.5.6.7:22 to "SSH-2.0-WS_FTP-SSH_7.7"
2020/06/25 07:50:58 DEBUG : fs cache: renaming cache item "./test.csv" to be canonical "/home/ak_zuar01/test.csv"
2020/06/25 07:50:58 ERROR : : error reading source directory: directory not found
2020/06/25 07:50:58 DEBUG : Local file system at /home/ak_zuar01/test.csv: Waiting for checks to finish
2020/06/25 07:50:58 DEBUG : Local file system at /home/ak_zuar01/test.csv: Waiting for transfers to finish
2020/06/25 07:50:58 INFO  : There was nothing to transfer
2020/06/25 07:50:58 ERROR : Attempt 1/1 failed with 1 errors and: directory not found
2020/06/25 07:50:58 INFO  : 
Transferred:   	         0 / 0 Bytes, -, 0 Bytes/s, ETA -
Errors:                 1 (retrying may help)
Elapsed time:         0.0s

2020/06/25 08:15:33 DEBUG : 12 go routines active
2020/06/25 08:15:33 Failed to copyto: directory not found
2

same exact error on another machine running 1.52.2 connecting to a totally different sftp server.

$ rclone copy nohup.out dddOutgoing:IN
2020/06/25 13:34:28 ERROR : nohup.out: Failed to copy: Put mkParentDir failed: mkdir "IN" failed: sftp: "Folder already exists: IN" (SSH_FX_FAILURE)
2020/06/25 13:34:28 ERROR : Attempt 1/3 failed with 1 errors and: Put mkParentDir failed: mkdir "IN" failed: sftp: "Folder already exists: IN" (SSH_FX_FAILURE)
2020/06/25 13:34:28 ERROR : nohup.out: Failed to copy: Put mkParentDir failed: mkdir "IN" failed: sftp: "Folder already exists: IN" (SSH_FX_FAILURE)
2020/06/25 13:34:28 ERROR : Attempt 2/3 failed with 1 errors and: Put mkParentDir failed: mkdir "IN" failed: sftp: "Folder already exists: IN" (SSH_FX_FAILURE)
2020/06/25 13:34:28 ERROR : nohup.out: Failed to copy: Put mkParentDir failed: mkdir "IN" failed: sftp: "Folder already exists: IN" (SSH_FX_FAILURE)
2020/06/25 13:34:28 ERROR : Attempt 3/3 failed with 1 errors and: Put mkParentDir failed: mkdir "IN" failed: sftp: "Folder already exists: IN" (SSH_FX_FAILURE)
2020/06/25 13:34:28 Failed to copy: Put mkParentDir failed: mkdir "IN" failed: sftp: "Folder already exists: IN" (SSH_FX_FAILURE)

what am i doing wrong??

3

I've tried use_insecure_cypher = true and disable_hashcheck = true .. no change.

any help would be greatly appreciated.

4

both ftp servers are similar versions:

SSH-2.0-WS_FTP-SSH_7.6.2

and

SSH-2.0-WS_FTP-SSH_7.7
5

This is indicating that the source directory can't be found

If you start a path with a leading / on the sftp remote then it means start from the root which is probably not what you want.

If that file is in the user's home directory then you want

"eleadtrack_old:Hendrick_SourceSubsource_USED_06252020.csv"

I'm not 100% sure what is going on here. Rclone attempted to create the IN directory (which is what it normally does) but the server replied "Folder already exists: IN" (SSH_FX_FAILURE) which also seems fair enough. However rclone should have checked that directory did exist before trying to create it, so it looks like that check failed for some reason.

If the server is a Windows server, then maybe the case of the directory is different - that might make a difference. So you could try In and in to see if that works.

This will show you what rclone sees

rclone lsf dddOutgoing:
6

Hey thanks for the reply.

The only similarity i can see is both of these are w_ftp servers of similar versions.

SSH-2.0-WS_FTP-SSH_7.7
# and ...
SSH-2.0-WS_FTP-SSH_7.62

On this particular server there are no directories beyond the root. if i rclone lsf, it just lists a bunch of csv files. I get the same results with or without the /

$ rclone -vv copyto eleadtrack:Hendrick_SourceSubsource_USED_06252020.csv ./text.csv
2020/06/25 14:10:27 DEBUG : rclone: Version "v1.49.0" starting with parameters ["rclone" "-vv" "copyto" "eleadtrack:Hendrick_SourceSubsource_USED_06252020.csv" "./text.csv"]
2020/06/25 14:10:27 DEBUG : Using config file from "/home/ak_zuar01/.config/rclone/rclone.conf"
2020/06/25 14:10:27 DEBUG : sftp://user@example.com:22/Hendrick_SourceSubsource_USED_06252020.csv: New connection 1.2.3.4:44066->4.5.6.7:22 to "SSH-2.0-WS_FTP-SSH_7.7"
2020/06/25 14:10:27 Failed to create file system for "eleadtrack:Hendrick_SourceSubsource_USED_06252020.csv": stat failed: sftp: "Folder not found: Hendrick_SourceSubsource_USED_06252020.csv" (SSH_FX_FAILURE)

$ rclone --low-level-retries 1 --retries 1 -vv copy test.txt eleadtrack:
2020/06/25 14:14:18 DEBUG : rclone: Version "v1.49.0" starting with parameters ["rclone" "--low-level-retries" "1" "--retries" "1" "-vv" "copy" "test.txt" "eleadtrack:"]
2020/06/25 14:14:18 DEBUG : Using config file from "/home/ak_zuar01/.config/rclone/rclone.conf"
2020/06/25 14:14:18 DEBUG : sftp://user@example.com:22/: New connection 1.2.3.4:44074->4.5.6.7:22 to "SSH-2.0-WS_FTP-SSH_7.7"
2020/06/25 14:14:19 ERROR : : error reading source directory: directory not found
2020/06/25 14:14:19 INFO  : sftp://user@example.com:22/: Waiting for checks to finish
2020/06/25 14:14:19 INFO  : sftp://user@example.com:22/: Waiting for transfers to finish
2020/06/25 14:14:19 ERROR : Attempt 1/1 failed with 2 errors and: directory not found
2020/06/25 14:14:19 Failed to copy with 2 errors: last error was: directory not found
7

on this server the folders IN and OUT exist. and i was attempting to write to IN. I tried In and in as well with the same results.

more efforts include:

$ rclone lsl dddOutgoing:
2020/06/25 15:34:04 ERROR : OUT: error listing: List failed: dirExists stat failed: sftp: "Folder not found: OUT" (SSH_FX_FAILURE)
2020/06/25 15:34:05 ERROR : IN: error listing: directory not found
2020/06/25 15:34:05 Failed to lsl with 3 errors: last error was: directory not found

$ rclone lsf dddOutgoing:/IN
2020/06/25 15:35:02 ERROR : : error listing: directory not found
2020/06/25 15:35:02 Failed to lsf with 2 errors: last error was: error in ListJSON: directory not found
8

What this is saying is that rclone asked for the details of "Hendrick_SourceSubsource_USED_06252020.csv" and the server replied "Folder not found" which is a bit odd.

In fact all the oddities seem to be due to stat not working as rclone expects.

It would be worth trying rclone 1.52.2 which was released recently to see if that is any different.

If that doesn't work can you try this which has a bit more debugging in which should hopefully help see what is going on - thanks!

https://beta.rclone.org/branch/v1.52.2-134-ga70bf051-fix-sftp-wsftp-beta/ (uploaded in 15-30 mins)

9

Morning. We have tried 1.52.2 on both servers. dddOutgoing output above was 1.52.2

tried a few things with the beta you linked to:

$ rclone version
rclone v1.52.2-134-ga70bf051-fix-sftp-wsftp-beta
- os/arch: linux/amd64
- go version: go1.14.4

$ rclone -vv lsd dddOutgoing:
2020/06/26 08:28:20 DEBUG : rclone: Version "v1.52.2-134-ga70bf051-fix-sftp-wsftp-beta" starting with parameters ["rclone" "-vv" "lsd" "dddOutgoing:"]
2020/06/26 08:28:20 DEBUG : Using config file from "/home/ubuntu/.config/rclone/rclone.conf"
2020/06/26 08:28:21 DEBUG : sftp://user@example.com:22/: New connection 1.2.3.4:36218->4.5.6.7:22 to "SSH-2.0-WS_FTP-SSH_7.6.2"
2020/06/26 08:28:21 DEBUG : Dir Stat result for "." is &sftp.fileInfo{name:".", size:0, mode:0x800001e8, mtime:time.Time{wall:0x0, ext:63728709547, loc:(*time.Location)(0x2a053a0)}, sys:(*sftp.FileStat)(0xc00003a280)}, <nil>
          -1 2020-06-08 23:47:31        -1 IN
          -1 2020-06-08 23:47:39        -1 OUT
2020/06/26 08:28:21 DEBUG : 11 go routines active

$ rclone -vv lsl dddOutgoing:
2020/06/26 08:31:47 DEBUG : rclone: Version "v1.52.2-134-ga70bf051-fix-sftp-wsftp-beta" starting with parameters ["rclone" "-vv" "lsl" "dddOutgoing:"]
2020/06/26 08:31:47 DEBUG : Using config file from "/home/ubuntu/.config/rclone/rclone.conf"
2020/06/26 08:31:47 DEBUG : sftp://user@example.com:22/: New connection 1.2.3.4:36222->4.5.6.7:22 to "SSH-2.0-WS_FTP-SSH_7.6.2"
2020/06/26 08:31:47 DEBUG : Dir Stat result for "." is &sftp.fileInfo{name:".", size:0, mode:0x800001e8, mtime:time.Time{wall:0x0, ext:63728709547, loc:(*time.Location)(0x2a053a0)}, sys:(*sftp.FileStat)(0xc00003b4c0)}, <nil>
2020/06/26 08:31:47 DEBUG : Dir Stat result for "OUT" is <nil>, &sftp.StatusError{Code:0x4, msg:"Folder not found: OUT", lang:""}
2020/06/26 08:31:47 ERROR : OUT: error listing: List failed: dirExists stat failed: sftp: "Folder not found: OUT" (SSH_FX_FAILURE)
2020/06/26 08:31:48 DEBUG : sftp://user@example.com:22/: New connection 1.2.3.4:36224->4.5.6.7:22 to "SSH-2.0-WS_FTP-SSH_7.6.2"
2020/06/26 08:31:48 DEBUG : Dir Stat result for "IN" is <nil>, &errors.errorString{s:"file does not exist"}
2020/06/26 08:31:48 ERROR : IN: error listing: directory not found
2020/06/26 08:31:48 DEBUG : 20 go routines active
2020/06/26 08:31:48 Failed to lsl with 3 errors: last error was: directory not found

$ rclone --low-level-retries 1 --retries 1 -vv copy nohup.out dddOutgoing:IN
2020/06/26 08:28:06 DEBUG : rclone: Version "v1.52.2-134-ga70bf051-fix-sftp-wsftp-beta" starting with parameters ["rclone" "--low-level-retries" "1" "--retries" "1" "-vv" "copy" "nohup.out" "dddOutgoing:IN"]
2020/06/26 08:28:06 DEBUG : Using config file from "/home/ubuntu/.config/rclone/rclone.conf"
2020/06/26 08:28:06 DEBUG : fs cache: renaming cache item "nohup.out" to be canonical "/home/ubuntu"
2020/06/26 08:28:06 DEBUG : sftp://user@example.com:22/IN: New connection 1.2.3.4:36216->4.5.6.7:22 to "SSH-2.0-WS_FTP-SSH_7.6.2"
2020/06/26 08:28:07 DEBUG : Stat result for "IN" is <nil>, &errors.errorString{s:"file does not exist"}
2020/06/26 08:28:07 DEBUG : Stat result for "IN/nohup.out" is <nil>, &errors.errorString{s:"file does not exist"}
2020/06/26 08:28:07 DEBUG : nohup.out: Need to transfer - File not found at Destination
2020/06/26 08:28:07 DEBUG : Dir Stat result for "IN" is <nil>, &errors.errorString{s:"file does not exist"}
2020/06/26 08:28:07 ERROR : nohup.out: Failed to copy: Put mkParentDir failed: mkdir "IN" failed: sftp: "Folder already exists: IN" (SSH_FX_FAILURE)
2020/06/26 08:28:07 ERROR : Attempt 1/1 failed with 1 errors and: Put mkParentDir failed: mkdir "IN" failed: sftp: "Folder already exists: IN" (SSH_FX_FAILURE)
2020/06/26 08:28:07 INFO  : 
Transferred:   	         0 / 0 Bytes, -, 0 Bytes/s, ETA -
Errors:                 1 (retrying may help)
Elapsed time:         0.0s

2020/06/26 08:28:07 DEBUG : 11 go routines active
2020/06/26 08:28:07 Failed to copy: Put mkParentDir failed: mkdir "IN" failed: sftp: "Folder already exists: IN" (SSH_FX_FAILURE)

thanks again!

10

Thanks! Those are useful logs.

This shows that rclone can find the current directory and the two directories within it IN and OUT.

However this shows that the server can't return a stat result for the IN directory and the OUT directory.

Which is very strange!

A bit of searching lead me to this very similar problem

So apparently I diagnosed and fixed this very similar problem in the upstream library already!

I checked and rclone is using the patched library. However the problem is very similar...

OK here is a trial fix which probably won't work, but you never know! I tried prefixing the direcotyr paths with ./.

https://beta.rclone.org/branch/v1.52.2-137-g43c4109a-fix-sftp-wsftp-beta/ (uploaded in 15-30 mins)

Is there any way I could get access to the server? I'm guessing not, but if it would it would speed up this process! Email nick@craig-wood.com or PM me if you want to discuss further.

11

No sorry. It's a customer's server so i can't give out creds. Tried the latest beta you shared:

$ rclone version
rclone v1.52.2-137-g43c4109a-fix-sftp-wsftp-beta
- os/arch: linux/amd64
- go version: go1.14.4

doing the same three commands for consistency. Got slightly different results on lsl

$ rclone -vv lsd dddOutgoing:
2020/06/27 09:55:37 DEBUG : rclone: Version "v1.52.2-137-g43c4109a-fix-sftp-wsftp-beta" starting with parameters ["rclone" "-vv" "lsd" "dddOutgoing:"]
2020/06/27 09:55:37 DEBUG : Using config file from "/home/ubuntu/.config/rclone/rclone.conf"
2020/06/27 09:55:38 DEBUG : sftp://user@example.com:22/: New connection 1.2.3.4:40768->4.5.6.7:22 to "SSH-2.0-WS_FTP-SSH_7.6.2"
2020/06/27 09:55:39 DEBUG : Dir Stat result for "." is &sftp.fileInfo{name:".", size:0, mode:0x800001e8, mtime:time.Time{wall:0x0, ext:63728709547, loc:(*time.Location)(0x2a063a0)}, sys:(*sftp.FileStat)(0xc0001b7600)}, <nil>
          -1 2020-06-08 23:47:31        -1 IN
          -1 2020-06-08 23:47:39        -1 OUT
2020/06/27 09:55:40 DEBUG : 11 go routines active

$ rclone -vv lsl dddOutgoing:
2020/06/27 09:55:50 DEBUG : rclone: Version "v1.52.2-137-g43c4109a-fix-sftp-wsftp-beta" starting with parameters ["rclone" "-vv" "lsl" "dddOutgoing:"]
2020/06/27 09:55:50 DEBUG : Using config file from "/home/ubuntu/.config/rclone/rclone.conf"
2020/06/27 09:55:50 DEBUG : sftp://user@example.com:22/: New connection 1.2.3.4:40770->4.5.6.7:22 to "SSH-2.0-WS_FTP-SSH_7.6.2"
2020/06/27 09:55:50 DEBUG : Dir Stat result for "." is &sftp.fileInfo{name:".", size:0, mode:0x800001e8, mtime:time.Time{wall:0x0, ext:63728709547, loc:(*time.Location)(0x2a063a0)}, sys:(*sftp.FileStat)(0xc00003a640)}, <nil>
2020/06/27 09:55:51 DEBUG : Dir Stat result for "./OUT" is &sftp.fileInfo{name:"OUT", size:0, mode:0x800001e8, mtime:time.Time{wall:0x0, ext:63727278459, loc:(*time.Location)(0x2a063a0)}, sys:(*sftp.FileStat)(0xc00003bd00)}, <nil>
2020/06/27 09:55:51 ERROR : OUT: error listing: error listing "OUT": sftp: "Folder not found: OUT/" (SSH_FX_FAILURE)
2020/06/27 09:55:51 DEBUG : sftp://user@example.com:22/: New connection 1.2.3.4:40772->4.5.6.7:22 to "SSH-2.0-WS_FTP-SSH_7.6.2"
2020/06/27 09:55:51 DEBUG : Dir Stat result for "./IN" is &sftp.fileInfo{name:"IN", size:0, mode:0x800001e8, mtime:time.Time{wall:0x0, ext:63727278451, loc:(*time.Location)(0x2a063a0)}, sys:(*sftp.FileStat)(0xc000125300)}, <nil>
2020/06/27 09:55:51 ERROR : IN: error listing: error listing "IN": sftp: "Permission Denied! " (SSH_FX_PERMISSION_DENIED)
2020/06/27 09:55:51 DEBUG : 20 go routines active
2020/06/27 09:55:51 Failed to lsl with 3 errors: last error was: error listing "IN": sftp: "Permission Denied! " (SSH_FX_PERMISSION_DENIED)

$ rclone --low-level-retries 1 --retries 1 -vv copy nohup.out dddOutgoing:IN
2020/06/27 10:07:11 DEBUG : rclone: Version "v1.52.2-137-g43c4109a-fix-sftp-wsftp-beta" starting with parameters ["rclone" "--low-level-retries" "1" "--retries" "1" "-vv" "copy" "nohup.out" "dddOutgoing:IN"]
2020/06/27 10:07:11 DEBUG : Using config file from "/home/ubuntu/.config/rclone/rclone.conf"
2020/06/27 10:07:11 DEBUG : fs cache: renaming cache item "nohup.out" to be canonical "/home/ubuntu"
2020/06/27 10:07:11 DEBUG : sftp://user@example.com:22/IN: New connection 1.2.3.4:40780->4.5.6.7:22 to "SSH-2.0-WS_FTP-SSH_7.6.2"
2020/06/27 10:07:13 DEBUG : Stat result for "IN" is <nil>, &errors.errorString{s:"file does not exist"}
2020/06/27 10:07:13 DEBUG : Stat result for "IN/nohup.out" is <nil>, &errors.errorString{s:"file does not exist"}
2020/06/27 10:07:13 DEBUG : nohup.out: Need to transfer - File not found at Destination
2020/06/27 10:07:13 DEBUG : Dir Stat result for "./IN" is <nil>, &errors.errorString{s:"file does not exist"}
2020/06/27 10:07:13 ERROR : nohup.out: Failed to copy: Put mkParentDir failed: mkdir "IN" failed: sftp: "Folder already exists: IN" (SSH_FX_FAILURE)
2020/06/27 10:07:13 ERROR : Attempt 1/1 failed with 1 errors and: Put mkParentDir failed: mkdir "IN" failed: sftp: "Folder already exists: IN" (SSH_FX_FAILURE)
2020/06/27 10:07:13 INFO  : 
Transferred:   	         0 / 0 Bytes, -, 0 Bytes/s, ETA -
Errors:                 1 (retrying may help)
Elapsed time:         0.0s

2020/06/27 10:07:13 DEBUG : 11 go routines active
2020/06/27 10:07:13 Failed to copy: Put mkParentDir failed: mkdir "IN" failed: sftp: "Folder already exists: IN" (SSH_FX_FAILURE)

if there's something i can do on the server to test, lemme know. sftp works fine, get, put, etc.

$ sftp <USERNAME>@example.com
<USERNAME>@example.com's password: 
Connected to example.com.
sftp> ls
IN   OUT  
Connected to example.com.
sftp> cd IN
sftp> put nohup.out 
Uploading nohup.out to /user/IN/nohup.out
nohup.out                                                                                                                                                                                                        100%  187     8.3KB/s   00:00    
sftp> ls
nohup.out
12

Interesting. It looks like it would work if rclone just used ./IN to do the listing.

The put from your sftp log makes me wonder if it is using absolute paths internally.

Is there more debug for it you could turn on so we can see what it is doing?

13

All I have for either of these servers is the sftp user/pass. I don't have any deeper access, cannot ssh in.

Is there something i can do on the machine i'm connecting with other than sftp -vv? Added logging to sshd, but it doesn't offer much info.

14

That gives me an idea...

Does it work if you use dddOutgoing:/user and dddOutgoing:/user/IN

I'd temporarily forgotten rclone could use the absolute path.

If that works then this will probably work without the /user prefix.

v1.52.2-138-gff714654-fix-sftp-wsftp-beta on branch fix-sftp-wsftp (uploaded in 15-30 mins)

15

I dont see a linux amd64 option?

trying absolute paths with the current version, but no dice:

$ rclone version
rclone v1.52.2-137-g43c4109a-fix-sftp-wsftp-beta
- os/arch: linux/amd64
- go version: go1.14.4

$ rclone --low-level-retries 1 --retries 1 -vv copy nohup.out dddOutgoing:/user
2020/06/29 11:41:26 DEBUG : rclone: Version "v1.52.2-137-g43c4109a-fix-sftp-wsftp-beta" starting with parameters ["rclone" "--low-level-retries" "1" "--retries" "1" "-vv" "copy" "nohup.out" "dddOutgoing:/user"]
2020/06/29 11:41:26 DEBUG : Using config file from "/home/ubuntu/.config/rclone/rclone.conf"
2020/06/29 11:41:26 DEBUG : fs cache: renaming cache item "nohup.out" to be canonical "/home/ubuntu"
2020/06/29 11:41:28 DEBUG : Stat result for "/user" is &sftp.fileInfo{name:"user", size:0, mode:0x800001e8, mtime:time.Time{wall:0x0, ext:63728709547, loc:(*time.Location)(0x2a063a0)}, sys:(*sftp.FileStat)(0xc0004bfa00)}, <nil>
2020/06/29 11:41:28 DEBUG : Stat result for "/user/nohup.out" is <nil>, &errors.errorString{s:"file does not exist"}
2020/06/29 11:41:28 DEBUG : nohup.out: Need to transfer - File not found at Destination
2020/06/29 11:41:28 DEBUG : Dir Stat result for ".//user" is <nil>, &errors.errorString{s:"file does not exist"}
2020/06/29 11:41:28 ERROR : nohup.out: Failed to copy: Put mkParentDir failed: mkdir "/user" failed: sftp: "Folder already exists: /user" (SSH_FX_FAILURE)
2020/06/29 11:41:28 ERROR : Attempt 1/1 failed with 1 errors and: Put mkParentDir failed: mkdir "/user" failed: sftp: "Folder already exists: /user" (SSH_FX_FAILURE)
2020/06/29 11:41:28 INFO  : 
Transferred:   	         0 / 0 Bytes, -, 0 Bytes/s, ETA -
Errors:                 1 (retrying may help)
Elapsed time:         0.0s

2020/06/29 11:41:28 DEBUG : 12 go routines active
2020/06/29 11:41:28 Failed to copy: Put mkParentDir failed: mkdir "/user" failed: sftp: "Folder already exists: /user" (SSH_FX_FAILURE)

$ rclone --low-level-retries 1 --retries 1 -vv copy nohup.out dddOutgoing:/user/IN
2020/06/29 11:41:37 DEBUG : rclone: Version "v1.52.2-137-g43c4109a-fix-sftp-wsftp-beta" starting with parameters ["rclone" "--low-level-retries" "1" "--retries" "1" "-vv" "copy" "nohup.out" "dddOutgoing:/user/IN"]
2020/06/29 11:41:37 DEBUG : Using config file from "/home/ubuntu/.config/rclone/rclone.conf"
2020/06/29 11:41:37 DEBUG : fs cache: renaming cache item "nohup.out" to be canonical "/home/ubuntu"
2020/06/29 11:41:38 DEBUG : Stat result for "/user/IN" is &sftp.fileInfo{name:"IN", size:0, mode:0x800001e8, mtime:time.Time{wall:0x0, ext:63729033927, loc:(*time.Location)(0x2a063a0)}, sys:(*sftp.FileStat)(0xc00015ad40)}, <nil>
2020/06/29 11:41:38 DEBUG : Stat result for "/user/IN/nohup.out" is <nil>, &errors.errorString{s:"file does not exist"}
2020/06/29 11:41:38 DEBUG : nohup.out: Need to transfer - File not found at Destination
2020/06/29 11:41:38 DEBUG : Dir Stat result for ".//user/IN" is <nil>, &sftp.StatusError{Code:0x4, msg:"Folder not found: /user/IN/user/IN", lang:""}
2020/06/29 11:41:38 ERROR : nohup.out: Failed to copy: Put mkParentDir failed: mkdir dirExists failed: dirExists stat failed: sftp: "Folder not found: /user/IN/user/IN" (SSH_FX_FAILURE)
2020/06/29 11:41:38 ERROR : Attempt 1/1 failed with 1 errors and: Put mkParentDir failed: mkdir dirExists failed: dirExists stat failed: sftp: "Folder not found: /user/IN/user/IN" (SSH_FX_FAILURE)
2020/06/29 11:41:38 INFO  : 
Transferred:   	         0 / 0 Bytes, -, 0 Bytes/s, ETA -
Errors:                 1 (retrying may help)
Elapsed time:         0.0s

2020/06/29 11:41:38 DEBUG : 11 go routines active
2020/06/29 11:41:38 Failed to copy: Put mkParentDir failed: mkdir dirExists failed: dirExists stat failed: sftp: "Folder not found: /user/IN/user/IN" (SSH_FX_FAILURE)
16

Can you try the absolute test with this rclone

https://beta.rclone.org/branch/v1.52.2-134-ga70bf051-fix-sftp-wsftp-beta/

I think the previous experiment is interfering.

I'll rebuild the failed test and post a new link in a bit.

17

absolutes worked with rclone v1.52.2-134-ga70bf051-fix-sftp-wsftp-beta !!

$ rclone --low-level-retries 1 --retries 1 -vv copy nohup.out dddOutgoing:/user/IN
2020/06/29 13:41:01 DEBUG : rclone: Version "v1.52.2-134-ga70bf051-fix-sftp-wsftp-beta" starting with parameters ["rclone" "--low-level-retries" "1" "--retries" "1" "-vv" "copy" "nohup.out" "dddOutgoing:/user/IN"]
2020/06/29 13:41:01 DEBUG : Using config file from "/home/ubuntu/.config/rclone/rclone.conf"
2020/06/29 13:41:01 DEBUG : fs cache: renaming cache item "nohup.out" to be canonical "/home/ubuntu"
2020/06/29 13:41:01 DEBUG : nohup.out: Need to transfer - File not found at Destination
2020/06/29 13:41:02 INFO  : nohup.out: Copied (new)
2020/06/29 13:41:02 INFO  : 
Transferred:   	       187 / 187 Bytes, 100%, 1.057 kBytes/s, ETA 0s
Transferred:            1 / 1, 100%
Elapsed time:         0.1s

2020/06/29 13:41:02 DEBUG : 11 go routines active
18

Whoo! We are getting somewhere.

So it looks like the WS FTP server has a problem with relative paths.

I could do a server quirk for it.

I think we need to take a look at what the sftp command does. Can you try your sftp session with -vvvvv and see if it logs anything about the sftp commands used?

19

I was looking for this earlier.. needed more v's. here you go:

sftp> ls
debug3: Sending SSH2_FXP_READDIR I:3
debug3: Received reply T:104 I:3
debug3: Received 4 SSH2_FXP_NAME responses
debug3: Sending SSH2_FXP_READDIR I:4
debug3: Received reply T:101 I:4
debug3: Received SSH2_FXP_STATUS 1
debug3: Sent message SSH2_FXP_CLOSE I:5
debug3: SSH2_FXP_STATUS 0
IN   OUT

sftp> cd IN/
debug3: Sent message fd 3 T:16 I:11
debug3: SSH_FXP_REALPATH /user/IN/ -> /user/IN size 0
debug3: Sent message fd 3 T:17 I:12
debug3: Received stat reply T:105 I:12

sftp> put nohup.out 
debug3: Looking up nohup.out
Uploading nohup.out to /user/IN/nohup.out
debug3: Sent message SSH2_FXP_OPEN I:17 P:/user/IN/nohup.out

nohup.out                                                                                                                                                                                                          0%    0     0.0KB/s   --:-- ETAdebug3: Sent message SSH2_FXP_WRITE I:18 O:0 S:187
debug3: SSH2_FXP_STATUS 0
debug3: In write loop, ack for 18 187 bytes at 0
nohup.out                                                                                                                                                                                                        100%  187     7.6KB/s   00:00    
debug3: Sent message SSH2_FXP_CLOSE I:18
debug3: SSH2_FXP_STATUS 0
20

Interesting, so the sftp command line client does use absolute paths everywhere - that is what the SSH_FXP_REALPATH code does.

I think that is probably a good enough reason for rclone to do the same. Can you have a go with this (assuming it builds properly!)

v1.52.2-147-g7bc35801-fix-sftp-wsftp-beta on branch fix-sftp-wsftp (uploaded in 15-30 mins)