How to configure an authenticated proxy in an SFTP backend client?

What is the problem you are having with rclone?

Failed to use SFTP backend when the client requires a HTTP proxy with auth

Run the command 'rclone version' and share the full output of the command.

rclone v1.65.0
- os/version: centos 7.6.1810 (64 bit)
- os/kernel: 3.10.0-957.el7.x86_64 (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.21.4
- go/linking: static
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

No. SFTP

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone lsd remote:

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

[aliyun]
type = sftp
ssh = /usr/bin/ssh -o "ProxyCommand=corkscrew {IP} 33128 %h %p ~/codebase/auth.txt"  {username}@{IP} -p 30032  -i ~/aliyun.pem
shell_type = unix
md5sum_command = md5sum
sha1sum_command = sha1sum

A log from the command that you were trying to run with the -vv flag

2024/01/16 18:14:56 DEBUG : rclone: Version "v1.65.0" starting with parameters ["rclone" "lsd" "aliyun:" "-vv"]
2024/01/16 18:14:56 DEBUG : Creating backend with remote "aliyun:"
2024/01/16 18:14:56 DEBUG : Using config file from "/mnt/petrelfs/{username}/.config/rclone/rclone.conf"
2024/01/16 18:14:56 DEBUG : sftp://{username}@:22/: ssh external: creating additional session
2024/01/16 18:14:56 DEBUG : sftp://{username}@:22/: ssh external: running: /usr/bin/ssh -o "ProxyCommand=corkscrew {IP} 33128 %h %p ~/codebase/auth.txt"  {username}@{IP} -p 30032  -i ~/aliyun.pem -s sftp
2024/01/16 18:14:56 DEBUG : pacer: low level retry 1/10 (error couldn't initialise SFTP: error receiving version packet from server: server unexpectedly closed connection: unexpected EOF)
2024/01/16 18:14:56 DEBUG : pacer: Rate limited, increasing sleep to 200ms
2024/01/16 18:14:56 DEBUG : sftp://{username}@:22/: ssh external: creating additional session
2024/01/16 18:14:56 DEBUG : sftp://{username}@:22/: ssh external: running: /usr/bin/ssh -o "ProxyCommand=corkscrew {IP} 33128 %h %p ~/codebase/auth.txt"  {username}@{IP} -p 30032  -i ~/aliyun.pem -s sftp
2024/01/16 18:14:56 DEBUG : pacer: low level retry 2/10 (error couldn't initialise SFTP: error receiving version packet from server: server unexpectedly closed connection: unexpected EOF)
2024/01/16 18:14:56 DEBUG : pacer: Rate limited, increasing sleep to 400ms
2024/01/16 18:14:56 DEBUG : sftp://{username}@:22/: ssh external: creating additional session
2024/01/16 18:14:56 DEBUG : sftp://{username}@:22/: ssh external: running: /usr/bin/ssh -o "ProxyCommand=corkscrew {IP} 33128 %h %p ~/codebase/auth.txt"  {username}@{IP} -p 30032  -i ~/aliyun.pem -s sftp
2024/01/16 18:14:56 DEBUG : pacer: low level retry 3/10 (error couldn't initialise SFTP: error receiving version packet from server: server unexpectedly closed connection: unexpected EOF)
2024/01/16 18:14:56 DEBUG : pacer: Rate limited, increasing sleep to 800ms
2024/01/16 18:14:57 DEBUG : sftp://{username}@:22/: ssh external: creating additional session
2024/01/16 18:14:57 DEBUG : sftp://{username}@:22/: ssh external: running: /usr/bin/ssh -o "ProxyCommand=corkscrew {IP} 33128 %h %p ~/codebase/auth.txt"  {username}@{IP} -p 30032  -i ~/aliyun.pem -s sftp
2024/01/16 18:14:57 DEBUG : pacer: low level retry 4/10 (error couldn't initialise SFTP: error receiving version packet from server: server unexpectedly closed connection: unexpected EOF)
2024/01/16 18:14:57 DEBUG : pacer: Rate limited, increasing sleep to 1.6s
2024/01/16 18:14:57 DEBUG : sftp://{username}@:22/: ssh external: creating additional session
2024/01/16 18:14:57 DEBUG : sftp://{username}@:22/: ssh external: running: /usr/bin/ssh -o "ProxyCommand=corkscrew {IP} 33128 %h %p ~/codebase/auth.txt"  {username}@{IP} -p 30032  -i ~/aliyun.pem -s sftp
2024/01/16 18:14:57 DEBUG : pacer: low level retry 5/10 (error couldn't initialise SFTP: error receiving version packet from server: server unexpectedly closed connection: unexpected EOF)
2024/01/16 18:14:57 DEBUG : pacer: Rate limited, increasing sleep to 2s
2024/01/16 18:14:59 DEBUG : sftp://{username}@:22/: ssh external: creating additional session
2024/01/16 18:14:59 DEBUG : sftp://{username}@:22/: ssh external: running: /usr/bin/ssh -o "ProxyCommand=corkscrew {IP} 33128 %h %p ~/codebase/auth.txt"  {username}@{IP} -p 30032  -i ~/aliyun.pem -s sftp
2024/01/16 18:14:59 DEBUG : pacer: low level retry 6/10 (error couldn't initialise SFTP: error receiving version packet from server: server unexpectedly closed connection: unexpected EOF)
2024/01/16 18:15:01 DEBUG : sftp://{username}@:22/: ssh external: creating additional session
2024/01/16 18:15:01 DEBUG : sftp://{username}@:22/: ssh external: running: /usr/bin/ssh -o "ProxyCommand=corkscrew {IP} 33128 %h %p ~/codebase/auth.txt"  {username}@{IP} -p 30032  -i ~/aliyun.pem -s sftp
2024/01/16 18:15:01 DEBUG : pacer: low level retry 7/10 (error couldn't initialise SFTP: error receiving version packet from server: server unexpectedly closed connection: unexpected EOF)
2024/01/16 18:15:03 DEBUG : sftp://{username}@:22/: ssh external: creating additional session
2024/01/16 18:15:03 DEBUG : sftp://{username}@:22/: ssh external: running: /usr/bin/ssh -o "ProxyCommand=corkscrew {IP} 33128 %h %p ~/codebase/auth.txt"  {username}@{IP} -p 30032  -i ~/aliyun.pem -s sftp
2024/01/16 18:15:03 DEBUG : pacer: low level retry 8/10 (error couldn't initialise SFTP: error receiving version packet from server: server unexpectedly closed connection: unexpected EOF)
2024/01/16 18:15:05 DEBUG : sftp://{username}@:22/: ssh external: creating additional session
2024/01/16 18:15:05 DEBUG : sftp://{username}@:22/: ssh external: running: /usr/bin/ssh -o "ProxyCommand=corkscrew {IP} 33128 %h %p ~/codebase/auth.txt"  {username}@{IP} -p 30032  -i ~/aliyun.pem -s sftp
2024/01/16 18:15:05 DEBUG : pacer: low level retry 9/10 (error couldn't initialise SFTP: error receiving version packet from server: server unexpectedly closed connection: unexpected EOF)
2024/01/16 18:15:07 DEBUG : sftp://{username}@:22/: ssh external: creating additional session
2024/01/16 18:15:07 DEBUG : sftp://{username}@:22/: ssh external: running: /usr/bin/ssh -o "ProxyCommand=corkscrew {IP} 33128 %h %p ~/codebase/auth.txt"  {username}@{IP} -p 30032  -i ~/aliyun.pem -s sftp
2024/01/16 18:15:07 DEBUG : pacer: low level retry 10/10 (error couldn't initialise SFTP: error receiving version packet from server: server unexpectedly closed connection: unexpected EOF)
2024/01/16 18:15:07 Failed to create file system for "aliyun:": NewFs: couldn't initialise SFTP: error receiving version packet from server: server unexpectedly closed connection: unexpected EOF

introdutction

Currently, I have an accessible SFTP service and can access it using rclone CLI on regular devices. However, I now have a device with restricted network access, which requires a proxy to access the internet. Moreover, the proxy requires authentication with a username and password.

I checked the official rclone documentation and found that it's possible to configure a socket proxy, but it seems like it doesn't support authentication with a username and password. So, I used an SSH parameter instead:

But this resulted in the following logs on the client:

2024/01/16 17:22:47 DEBUG : rclone: Version "v1.65.0" starting with parameters ["rclone" "lsd" "aliyun:" "-vvv"]
2024/01/16 17:22:47 DEBUG : Creating backend with remote "aliyun:"
2024/01/16 17:22:47 DEBUG : Using config file from "/mnt/petrelfs/yehaochen/.config/rclone/rclone.conf"
2024/01/16 17:22:47 DEBUG : sftp://yehaochen@:22/: ssh external: creating additional session
2024/01/16 17:22:47 DEBUG : sftp://yehaochen@:22/: ssh external: running: /usr/bin/ssh -o "ProxyCommand=corkscrew 10.1.8.50 33128 %h %p ~/codebase/auth.txt"  yehaochen@8.130.120.199 -p 30032  -i ~/aliyun.pem & -s sftp
2024/01/16 17:22:47 DEBUG : pacer: low level retry 1/10 (error couldn't initialise SFTP: error receiving version packet from server: server unexpectedly closed connection: unexpected EOF)
...
2024/01/16 17:22:48 DEBUG : pacer: low level retry 5/10 (error couldn't initialise SFTP: error receiving version packet from server: server unexpectedly closed connection: unexpected EOF)
2024/01/16 17:22:48

DEBUG : pacer: Rate limited, increasing sleep to 2s

The SSH server logs are as follows:

Accepted publickey for yehaochen from 10.1.112.52 port 34764 ssh2: RSA SHA256:TxolQfi6UZrG/RqER7MKYbCCxVt5hO2n+Egw3JAy624
debug1: monitor_child_preauth: yehaochen has been authenticated by privileged process
debug1: monitor_read_log: child log fd closed
debug1: PAM: establishing credentials
User child is on pid 160450
debug1: SELinux support disabled
debug1: PAM: establishing credentials
debug1: permanently_set_uid: 10555/10555
debug1: rekey after 134217728 blocks
debug1: rekey after 134217728 blocks
debug1: ssh_packet_set_postauth: called
debug1: Entering interactive session for SSH2.
debug1: server_init_dispatch
debug1: server_input_channel_open: ctype session rchan 0 win 2097152 max 32768
debug1: input_session_request
debug1: channel 0: new [server-session]
debug1: session_new: session 0
debug1: session_open: channel 0
debug1: session_open: session 0: link with channel 0
debug1: server_input_channel_open: confirm session
debug1: server_input_global_request: rtype no-more-sessions@openssh.com want_reply 0
debug1: server_input_channel_req: channel 0 request subsystem reply 1
debug1: session_by_channel: session 0 channel 0
debug1: session_input_channel_req: session 0 req subsystem
debug1: subsystem: exec() /usr/lib/openssh/sftp-server
Starting session: subsystem 'sftp' for yehaochen from 10.1.112.52 port 34764 id 0

It seems that the SSH request is processed normally, but it gets stuck on the client side.

Further verification showed that rclone appears to append the -s sftp parameter to the ssh command. Strangely, when this parameter is added, the ssh command indeed gets stuck and doesn't enter an interactive mode or anything similar.

I'm not very familiar with the specifics of rclone and SFTP, so I would like to ask what should I do? Is the use of the ssh parameter in the configuration file correct? And can rclone's SFTP configuration handle a proxy with authentication?

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.