caduceus
April 30, 2019, 5:08pm
1
I connected a remote SFTP source, which I am able to ls just fine. But when I try to sync or copy, I receive an error as rclone attempts to transfer each file. I'm running on OSX, and I believe the remote server is running on a Windows system win win_scp..
I am getting error like this:
2019/04/30 12:13:27 NOTICE: aaaaa_valley_hhhhhh_ccp/CCP_Valley_hhhhhh_190429.csv: Removing partially written file on error: sftp: "Folder not found: c:\ftpsites\aiep\valley_hhhhhh\CCP_Valley_hhhhhh_190429.csv" (SSH_FX_FAILURE)
2019/04/30 12:13:27 ERROR : rclone sync aiep:./ ~/home/Download/PEIA_CCP
2019/04/30 12:13:27 NOTICE: aaaaaaa_valley_hhhhhh_ccp/CCP_Valley_hhhhhh_190429.csv: Removing partially written file on error: sftp: "Folder not found: c:\ftpsites\aiep\valley_hhhhhh\CCP_Valley_hhhhhh_190429.csv" (SSH_FX_FAILURE)
2019/04/30 12:13:27 ERROR : aaaaaaa_valley_hhhhhh_ccp/CCP_Valley_hhhhhh_190429.csv: Failed to copy: sftp: "Folder not found: c:\ftpsites\aiep\valley_health\CCP_Valley_hhhhhh_190429.csv" (SSH_FX_FAILURE)
debug with -vv
rclone copy aiep:./ ~/home/Download/aiep_CCP -vv
rclone version shows what?
if you run the same command with -vv on it, can you share those results?
caduceus
April 30, 2019, 5:18pm
3
rclone version
rclone v1.47.0
os/arch: darwin/amd64
go version: go1.12.4
I added the debug option info to the original post.
What does your rclone.conf look like for that? I just did a test on my Mac to a Linux host with SFTP and copied a few files over.
caduceus
April 30, 2019, 6:16pm
5
Hi Animostiy022,
It looks like this:
type = sftp
host = ftp.thehost.com
user = User.name
port = 22
pass = {password}
use_insecure_cipher = true
disable_hashcheck = false
I had previously not had disable_hashcheck , but it did the same thing.
I have a pretty similar config:
[SFTP]
type = sftp
host = home.thehost.us
user = felix
port = 4022
pass = password
use_insecure_cipher = true
Can you do a ls on a single file that you know is there like:
rclone ls -vv aiep: aaaaaaa_iiiiicreek_ccp/CCP_iiiii_Creek_190429.csv
I think that is the path from reading the logs, but not quite sure.
And if you test via a scp command on the same host/path, can you share that too?
I just did a test file copy:
scp felix@home:hosts .
hosts
caduceus
April 30, 2019, 7:10pm
7
I can access the sftp server just fine from the command line on my mac and in other sftp clients… here are the results of the -vv on ls:
rclone ls -vv aiep:aaaaaa_valley_hhhhh_ccp/CCP_Valley_hhhhh_190429.csv
caduceus
April 30, 2019, 7:15pm
8
What was weird for me were these paths:
I don’t know where the c:\ftpsites came from…
So when you run that you get those errors?
What does rclone ls aiep: -vv do?
ncw
May 1, 2019, 9:23am
10
That is a strange error… It makes me wonder if rclone is mis-identifying files as directories…
That didn’t print the file did it? So it looks like rclone tried to list it as a directory.
Are these files (on the server) maybe symlinks (or junction points) on windows - that might explain what is going on.
Which server is it?
caduceus
May 1, 2019, 1:43pm
11
Animosity022:
rclone ls aiep: -vv
rclone ls aiep: -vv
caduceus
May 1, 2019, 1:44pm
12
No it did not print the file. I pasted the entire output.
I don’t know if the files are symlinks on Windows, looking at the error it was my first thought, having looked through other error reports and posts in forums, etc.
The server is an associate company where I get files waiting for me to process. I believe the server is Win_SCP as I had once tried to ssh to it instead of SFTP and it gave an error , WinSCP: this is end-of-file:0 when I pressed enter. I have no issues when connecting via SFTP.
ncw
May 1, 2019, 7:55pm
13
caduceus:
rclone ls aiep: -vv
Could you repeat that with lsf -R instead so rclone lsf -R aiep: -vv and post the output? That will tell us whether rclone things they are files or directories.
caduceus:
I don’t know if the files are symlinks on Windows, looking at the error it was my first thought, having looked through other error reports and posts in forums, etc.
I don’t know if a different SFTP client might tell you.
Ah OK!
caduceus
May 1, 2019, 11:15pm
14
rclone lsf -R aiep: -vv
caduceus
May 1, 2019, 11:25pm
15
I did a debug on sftp from command line:
debug1: Local version string SSH-2.0-OpenSSH_7.9
debug2: compression ctos: none,zlib@openssh.com ,zlibnone,zlib@openssh.com ,zlib
ncw
May 2, 2019, 8:07am
16
caduceus:
rclone lsf -R aiep: -vv
That looks sensible…
After tracing through the code, I think what is happening is when the sftp library attempts to read from the file it gets the error sftp: “Folder not found: c:\ftpsites\aiep\iiiiicreek\CCP_iiiii_Creek_190429.csv” (SSH_FX_FAILURE)
From the listings, rclone thinks it is a file, but when the upstream library reads it things it is a folder, or the server thinks it is a folder.
Could you try doing this from a command line sftp client on the “aiep_CCP” directory? I used the one that comes with ubuntu
$ sftp sftptest@localhost
Connected to sftptest@localhost.
sftp> ls -l
lrwxrwxrwx 1 root root 13 Jan 28 17:08 dangling_link
-rw-r--r-- 1 sftptest sftptest 6 Dec 30 16:13 hello[brackends].txt
drwxrwxr-x 7 sftptest sftptest 4096 Nov 28 09:58 restic
drwxr-xr-x 2 sftptest sftptest 4096 Dec 30 16:13 subdir
drwxr-xr-x 3 sftptest sftptest 4096 Dec 30 16:51 test
That should show exactly what type of thing the sftp server thinks things are.
Is there any chance I could get access to the sftp server? I’m guessing not, but if it is OK then please DM me!
caduceus
May 2, 2019, 11:38am
17
I didn’t see the aiep_CCP directory you were refering to, to I did an ls -l from the home/root where I login to and also inside the aaaaaaa_iiiiiicreek_ccp directory… I really appreciate all the help looking into this. Unfortunately, the server isn’t mine.
sftp> ls -l
sftp> ls -l
ncw
May 2, 2019, 3:19pm
18
Thanks for that, unfortunately those look normal!
I’ve compiled a version of the sftp library with debugging. This will print lots of hex numbers!
https://beta.rclone.org/branch/v1.47.0-032-g5c0574ea-fix-sftp-folder-beta/ (uploaded in 15-30 mins)
Can you find the minimum thing which prints the “Folder not found” error.
Something like this might do it
rclone copyto --low-level-retries 1 --retries 1 -vv aiep:aaaaaa_valley_hhhhh_ccp/CCP_Valley_hhhhh_190429.csv /tmp/z.csv
Then paste the debug stream!
Once we’ve got that I’ll probably ask for help upstream.
caduceus
May 2, 2019, 5:02pm
19
./rclone copyto --low-level-retries 1 --retries 1 -vv aiep:aaaaaaa_valley_hhhhh_ccp/CCP_Valley_hhhhh_190429.csv ~/Downloads/z.csv
ncw
May 7, 2019, 9:58am
20
Thank you for that - that is very helpful.
I opened an upstream issue to ask the SFTP experts exactly what it means!
1 Like