Regression in ws_ftp support

Suspected Bug
1

What is the problem you are having with rclone?

In 2020 a fix for ws_ftp was put in place. SSH_FX_FAILURE errors have reappeared in 1.59 on an SFTP endpoint that works fine in 1.58.

Run the command 'rclone version' and share the full output of the command.

rclone v1.59.0

  • os/version: alpine 3.16.0 (64 bit)
  • os/kernel: 3.10.0-1062.9.1.el7.x86_64 (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.18.3
  • go/linking: static
  • go/tags: none

Which cloud storage system are you using? (eg Google Drive)

SFTP

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone -vv --timeout 10s copyto sftpsource:directory/file.csv s3:bucket/prefix/file.csv

The rclone config contents with secrets removed.

[sftpsource]
type = sftp
host = sftp.domain.com
port = 22
user = username
pass = encryptedPassword
use_insecure_cipher = true
md5sum_command = none
sha1sum_command = none

A log from the command with the -vv flag

2022/07/18 19:59:23 DEBUG : rclone: Version "v1.59.0" starting with parameters ["rclone" "-vv" "--timeout" "10s" "copyto" "sftpsource:directory/file.csv" "s3:bucket/prefix/file.csv"]
2022/07/18 19:59:23 DEBUG : Creating backend with remote "sftpsource:directory/file.csv"
2022/07/18 19:59:23 DEBUG : Using config file from "/config/rclone/rclone.conf"
2022/07/18 19:59:26 DEBUG : sftp://username@sftp.domain.com:22/directory/file.csv: New connection IPADDRESS:41172->IPADDRESS:22 to "SSH-2.0-WS_FTP-SSH_7.5.1"
2022/07/18 19:59:26 DEBUG : sftp://username@sftp.domain.com:22/directory/file.csv: Running shell type detection remote command: echo ${ShellId}%ComSpec%
2022/07/18 19:59:26 DEBUG : sftp://username@sftp.domain.com:22/directory/file.csv: Remote command failed: ssh: command echo ${ShellId}%ComSpec% failed (stdout=[]) (stderr=[])
2022/07/18 19:59:26 DEBUG : sftp://username@sftp.domain.com:22/directory/file.csv: Shell type "unix" detected (set option shell_type to override)
2022/07/18 19:59:26 DEBUG : Saving config "shell_type" in section "sftpsource" of the config file
2022/07/18 19:59:26 DEBUG : Keeping previous permissions for config file: -rw-rw-r--
2022/07/18 19:59:26 DEBUG : sftp://username@sftp.domain.com:22/directory/file.csv: Failed to resolve path - using relative paths: sftp: "Folder not found: directory/file.csv" (SSH_FX_FAILURE)
2022/07/18 19:59:26 Failed to create file system for "sftpsource:directory/filename.csv": stat failed: sftp: "Folder not found: directory/filename.csv" (SSH_FX_FAILURE)

Output from previous version

2022/07/18 19:50:36 DEBUG : rclone: Version "v1.58.1" starting with parameters ["rclone" "-vvvv" "--timeout" "10s" "copyto" "sftpsource:directory/file.csv" "s3:bucket/prefix/file.csv"]
2022/07/18 19:50:36 DEBUG : Creating backend with remote "sftpsource:directory/file.csv"
2022/07/18 19:50:36 DEBUG : Using config file from "/config/rclone/rclone.conf"
2022/07/18 19:50:42 DEBUG : sftp://username@sftp.domain.com:22/directory/file.csv: New connection IPADDRESS:40694->IPADDRESS:22 to "SSH-2.0-WS_FTP-SSH_7.5.1"
2022/07/18 19:50:42 DEBUG : sftp://username@sftp.domain.com:22/directory/file.csv: Using absolute root directory "/users/USERNAME/directory/file.csv"
2022/07/18 19:50:42 DEBUG : Creating backend with remote "s3:bucket/prefix/file.csv"
2022/07/18 19:51:42 DEBUG : sftp://username@sftp.domain.com:22/directory/file.csv: closing 1 unused connections
2

I would guess this is something to do with your sftp changes @albertony?

@ddrinka is there any chance you could git bisect the problem?

3

I'm not set up to build here, just working with the Docker releases. But here's where I'd start looking:
sftp.go@b4091f2 Line 861

4

I have tried to make a fix, see PR sftp: fix issue with WS_FTP by working around failing RealPath by albertony · Pull Request #6324 · rclone/rclone · GitHub

Please test when beta build is ready at https://beta.rclone.org/branch/sftp-realpath-fix/

5

Your fix has resolved the issue for me.

2022/07/19 15:31:07 DEBUG : rclone: Version "v1.60.0-beta.6367.5a5100663.sftp-realpath-fix" starting with parameters ["rclone" "-vv" "--timeout" "10s" "copyto" "sftpsource:directory/file.csv" "s3:bucket/prefix/file.csv"]
2022/07/19 15:31:07 DEBUG : Creating backend with remote "sftpsource:directory/file.csv"
2022/07/19 15:31:07 DEBUG : Using config file from "/config/rclone/rclone.conf"
2022/07/19 15:31:11 DEBUG : sftp://username@sftp.domain.com:22/directory/file.csv: New connection IPADDRESS:54780->IPADDRESS:22 to "SSH-2.0-WS_FTP-SSH_7.5.1"
2022/07/19 15:31:11 DEBUG : sftp://username@sftp.domain.com:22/directory/file.csv: Running shell type detection remote command: echo ${ShellId}%ComSpec%
2022/07/19 15:31:11 DEBUG : sftp://username@sftp.domain.com:22/directory/file.csv: Remote command failed: ssh: command echo ${ShellId}%ComSpec% failed (stdout=[]) (stderr=[])
2022/07/19 15:31:11 DEBUG : sftp://username@sftp.domain.com:22/directory/file.csv: Shell type "unix" detected (set option shell_type to override)
2022/07/19 15:31:11 DEBUG : Saving config "shell_type" in section "sftpsource" of the config file
2022/07/19 15:31:11 DEBUG : Keeping previous permissions for config file: -rw-rw-r--
2022/07/19 15:31:12 DEBUG : sftp://username@sftp.domain.com:22/directory/file.csv: Failed to resolve path using RealPath: sftp: "Folder not found: directory/file.csv" (SSH_FX_FAILURE)
2022/07/19 15:31:12 DEBUG : sftp://username@sftp.domain.com:22/directory/file.csv: Relative path joined with current directory to get absolute path "/users/USERNAME/directory/file.csv"
2022/07/19 15:31:12 DEBUG : sftp://username@sftp.domain.com:22/directory/file.csv: Using root directory "/users/USERNAME/directory/file.csv"
2022/07/19 15:31:12 DEBUG : Creating backend with remote "s3:bucket/prefix/file.csv"
1 Like
6

Perfect, thank you for testing - and reporting back!

1 Like
7

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.