Can rclone be made to work with an SFTP server confining users to an SFTP Jail and no login?

justme · March 6, 2024, 8:25pm

What is the problem you are having with rclone?

For security reasons, users of an application I am working on use an SFTP server but are not allowed SSH login and are confined to an SFTP Jail. rclone gives me the error messages shown in the log.

Run the command 'rclone version' and share the full output of the command.

C:\Users\justme>rclone version
rclone v1.65.2
- os/version: Microsoft Windows 10 Home 22H2 (64 bit)
- os/kernel: 10.0.19045.4046 (x86_64)
- os/type: windows
- os/arch: amd64
- go/version: go1.21.6
- go/linking: static
- go/tags: cmount

C:\Users\justme>

Which cloud storage system are you using? (eg Google Drive)

None. I am using my own SFTP server on Ubuntu 22.04.

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

C:\Users\justme>rclone mount fort_tu1_new_2: Z:  
2024/03/06 10:45:23 NOTICE: fort_tu1_new_2: --sftp-ssh is in use - ignoring user/host/port from config - set in the parameters to --sftp-ssh (remove them from the config to silence this warning)
2024/03/06 10:46:45 Failed to create file system for "fort_tu1_new_2:": NewFs: couldn't initialise SFTP: error receiving version packet from server: server unexpectedly closed connection: unexpected EOF

C:\Users\justme>

Please run 'rclone config redacted' and share the full output. If you get

[fort_tu1]
type = sftp
host = 208.113.134.46
user = tu1
key_pem = C:\Users\justme\.ssh\id_ed25519_tu1.pem
ssh = ssh -i C:\Users\justme\.ssh\id_ed25519_tu1.pem tu1@208.113.134.46
known_hosts_file = C:\Users\justme\.ssh\known_hosts
shell_type = none
skip_links = true
idle_timeout = 10m0s

[fort_tu1_new]
type = sftp
host = 208.113.134.46
user = tu1
key_file = C:\Users\justme\.ssh\id_ed25519_tu1
ssh = ssh  -i C:\Users\justme\.ssh\id_ed25519_tu1 tu1@208.113.134.46
shell_type = none
skip_links = true
idle_timeout = 10m0s

[fort_tu1_new_2]
type = sftp
host = 208.113.134.46
user = tu1
key_file = C:\Users\justme\.ssh\id_ed25519_tu1
ssh = ssh  -i C:\Users\justme\.ssh\id_ed25519_tu1 tu1@208.113.134.46
idle_timeout = 10m0s

A log from the command that you were trying to run with the `-vv` flag

C:\Users\justme>rclone mount fort_tu1_new_2: Z:  -vv
2024/03/06 11:28:12 DEBUG : rclone: Version "v1.65.2" starting with parameters ["rclone" "mount" "fort_tu1_new_2:" "Z:" "-vv"]
2024/03/06 11:28:12 DEBUG : Creating backend with remote "fort_tu1_new_2:"
2024/03/06 11:28:12 DEBUG : Using config file from "C:\\Users\\justme\\Desktop\\rclone\\rclone.conf"
2024/03/06 11:28:12 NOTICE: fort_tu1_new_2: --sftp-ssh is in use - ignoring user/host/port from config - set in the parameters to --sftp-ssh (remove them from the config to silence this warning)
2024/03/06 11:28:12 DEBUG : sftp://tu1@208.113.134.46:22/: ssh external: creating additional session
2024/03/06 11:28:12 DEBUG : sftp://tu1@208.113.134.46:22/: ssh external: running: ssh  -i C:\Users\justme\.ssh\id_ed25519_tu1 tu1@208.113.134.46 -s sftp
2024/03/06 11:28:21 DEBUG : pacer: low level retry 1/10 (error couldn't initialise SFTP: error receiving version packet from server: server unexpectedly closed connection: unexpected EOF)
2024/03/06 11:28:21 DEBUG : pacer: Rate limited, increasing sleep to 200ms
2024/03/06 11:28:21 DEBUG : sftp://tu1@208.113.134.46:22/: ssh external: creating additional session
2024/03/06 11:28:21 DEBUG : sftp://tu1@208.113.134.46:22/: ssh external: running: ssh  -i C:\Users\justme\.ssh\id_ed25519_tu1 tu1@208.113.134.46 -s sftp
2024/03/06 11:28:29 DEBUG : pacer: low level retry 2/10 (error couldn't initialise SFTP: error receiving version packet from server: server unexpectedly closed connection: unexpected EOF)
2024/03/06 11:28:29 DEBUG : pacer: Rate limited, increasing sleep to 400ms
2024/03/06 11:28:29 DEBUG : sftp://tu1@208.113.134.46:22/: ssh external: creating additional session
2024/03/06 11:28:29 DEBUG : sftp://tu1@208.113.134.46:22/: ssh external: running: ssh  -i C:\Users\justme\.ssh\id_ed25519_tu1 tu1@208.113.134.46 -s sftp
2024/03/06 11:28:37 DEBUG : pacer: low level retry 3/10 (error couldn't initialise SFTP: error receiving version packet from server: server unexpectedly closed connection: unexpected EOF)
2024/03/06 11:28:37 DEBUG : pacer: Rate limited, increasing sleep to 800ms
2024/03/06 11:28:37 DEBUG : sftp://tu1@208.113.134.46:22/: ssh external: creating additional session
2024/03/06 11:28:37 DEBUG : sftp://tu1@208.113.134.46:22/: ssh external: running: ssh  -i C:\Users\justme\.ssh\id_ed25519_tu1 tu1@208.113.134.46 -s sftp
2024/03/06 11:28:45 DEBUG : pacer: low level retry 4/10 (error couldn't initialise SFTP: error receiving version packet from server: server unexpectedly closed connection: unexpected EOF)
2024/03/06 11:28:45 DEBUG : pacer: Rate limited, increasing sleep to 1.6s
2024/03/06 11:28:45 DEBUG : sftp://tu1@208.113.134.46:22/: ssh external: creating additional session
2024/03/06 11:28:45 DEBUG : sftp://tu1@208.113.134.46:22/: ssh external: running: ssh  -i C:\Users\justme\.ssh\id_ed25519_tu1 tu1@208.113.134.46 -s sftp
2024/03/06 11:28:53 DEBUG : pacer: low level retry 5/10 (error couldn't initialise SFTP: error receiving version packet from server: server unexpectedly closed connection: unexpected EOF)
2024/03/06 11:28:53 DEBUG : pacer: Rate limited, increasing sleep to 2s
2024/03/06 11:28:53 DEBUG : sftp://tu1@208.113.134.46:22/: ssh external: creating additional session
2024/03/06 11:28:53 DEBUG : sftp://tu1@208.113.134.46:22/: ssh external: running: ssh  -i C:\Users\justme\.ssh\id_ed25519_tu1 tu1@208.113.134.46 -s sftp
2024/03/06 11:29:02 DEBUG : pacer: low level retry 6/10 (error couldn't initialise SFTP: error receiving version packet from server: server unexpectedly closed connection: unexpected EOF)
2024/03/06 11:29:02 DEBUG : sftp://tu1@208.113.134.46:22/: ssh external: creating additional session
2024/03/06 11:29:02 DEBUG : sftp://tu1@208.113.134.46:22/: ssh external: running: ssh  -i C:\Users\justme\.ssh\id_ed25519_tu1 tu1@208.113.134.46 -s sftp
2024/03/06 11:29:10 DEBUG : pacer: low level retry 7/10 (error couldn't initialise SFTP: error receiving version packet from server: server unexpectedly closed connection: unexpected EOF)
2024/03/06 11:29:10 DEBUG : sftp://tu1@208.113.134.46:22/: ssh external: creating additional session
2024/03/06 11:29:10 DEBUG : sftp://tu1@208.113.134.46:22/: ssh external: running: ssh  -i C:\Users\justme\.ssh\id_ed25519_tu1 tu1@208.113.134.46 -s sftp
2024/03/06 11:29:18 DEBUG : pacer: low level retry 8/10 (error couldn't initialise SFTP: error receiving version packet from server: server unexpectedly closed connection: unexpected EOF)
2024/03/06 11:29:18 DEBUG : sftp://tu1@208.113.134.46:22/: ssh external: creating additional session
2024/03/06 11:29:18 DEBUG : sftp://tu1@208.113.134.46:22/: ssh external: running: ssh  -i C:\Users\justme\.ssh\id_ed25519_tu1 tu1@208.113.134.46 -s sftp
2024/03/06 11:29:26 DEBUG : pacer: low level retry 9/10 (error couldn't initialise SFTP: error receiving version packet from server: server unexpectedly closed connection: unexpected EOF)
2024/03/06 11:29:26 DEBUG : sftp://tu1@208.113.134.46:22/: ssh external: creating additional session
2024/03/06 11:29:26 DEBUG : sftp://tu1@208.113.134.46:22/: ssh external: running: ssh  -i C:\Users\justme\.ssh\id_ed25519_tu1 tu1@208.113.134.46 -s sftp
2024/03/06 11:29:34 DEBUG : pacer: low level retry 10/10 (error couldn't initialise SFTP: error receiving version packet from server: server unexpectedly closed connection: unexpected EOF)
2024/03/06 11:29:34 Failed to create file system for "fort_tu1_new_2:": NewFs: couldn't initialise SFTP: error receiving version packet from server: server unexpectedly closed connection: unexpected EOF

C:\Users\justme>

The SFTP server allows SFTP with no SSH login and users are in an SFTP Jail. This is what happens if the user tries to do an SSH login

C:\Users\justme>ssh  -i C:\Users\justme\.ssh\id_ed25519_tu1 tu1@208.113.134.46
This service allows sftp connections only.
Connection to 208.113.134.46 closed.

C:\Users\justme>

It looks to me like rclone needs to do ssh login. If that is the case then rclone will not work for my application. For security reasons, users are not allowed ssh login and are confined to an SFTP Jail. Can rclone working with SFTP jailed users be made to work without SSH login? Please advise.

justme · March 19, 2024, 7:41pm

Just to give this thread a proper ending, I found that to prevent the remote from attempting ssh login, the line "ssh = " in the config must not show an ssh command or else the remote will attempt ssh login even though there is "shell_type = none".

kapitainsky · March 19, 2024, 7:53pm

Would you mind to update rclone docs with this clarification? It is not something everybody experience but is priceless when needed IMO

I think it would be great to add this to to:

github.com

rclone/rclone/blob/master/backend/sftp/sftp.go

//go:build !plan9
// +build !plan9

// Package sftp provides a filesystem interface using github.com/pkg/sftp
package sftp

import (
	"bytes"
	"context"
	"errors"
	"fmt"
	"io"
	iofs "io/fs"
	"os"
	"path"
	"regexp"
	"strconv"
	"strings"
	"sync"
	"sync/atomic"

This file has been truncated. show original

here:

github.com

rclone/rclone/blob/7803b4ed6c3256f25270e3b3024011f8f1e77ca0/backend/sftp/sftp.go#L410-L438


      
          			Advanced: true,
          		}, {
          			Name:    "ssh",
          			Default: fs.SpaceSepList{},
          			Help: `Path and arguments to external ssh binary.
          
          Normally rclone will use its internal ssh library to connect to the
          SFTP server. However it does not implement all possible ssh options so
          it may be desirable to use an external ssh binary.
          
          Rclone ignores all the internal config if you use this option and
          expects you to configure the ssh binary with the user/host/port and
          any other options you need.
          
          **Important** The ssh command must log in without asking for a
          password so needs to be configured with keys or certificates.
          
          Rclone will run the command supplied either with the additional
          arguments "-s sftp" to access the SFTP subsystem or with commands such
          as "md5sum /path/to/file" appended to read checksums.

This file has been truncated. show original

click edit (pen icon) -> create fork -> edit -> submit PR

or maybe in "Limitations"..?

github.com

rclone/rclone/blob/master/docs/content/sftp.md#limitations

---
title: "SFTP"
description: "SFTP"
versionIntroduced: "v1.36"
---

# {{< icon "fa fa-server" >}} SFTP

SFTP is the [Secure (or SSH) File Transfer
Protocol](https://en.wikipedia.org/wiki/SSH_File_Transfer_Protocol).

The SFTP backend can be used with a number of different providers:

{{< provider_list >}}
{{< provider name="Hetzner Storage Box" home="https://www.hetzner.com/storage/storage-box" config="/sftp/#hetzner-storage-box">}}
{{< provider name="rsync.net" home="https://rsync.net/products/rclone.html" config="/sftp/#rsync-net">}}
{{< /provider_list >}}

SFTP runs over SSH v2 and is installed as standard with most modern
SSH installations.

This file has been truncated. show original

github.com

rclone/rclone/blob/7803b4ed6c3256f25270e3b3024011f8f1e77ca0/docs/content/sftp.md?plain=1#L1058-L1091


      
          ## Limitations
          
          On some SFTP servers (e.g. Synology) the paths are different
          for SSH and SFTP so the hashes can't be calculated properly.
          For them using `disable_hashcheck` is a good idea.
          
          The only ssh agent supported under Windows is Putty's pageant.
          
          The Go SSH library disables the use of the aes128-cbc cipher by
          default, due to security concerns. This can be re-enabled on a
          per-connection basis by setting the `use_insecure_cipher` setting in
          the configuration file to `true`. Further details on the insecurity of
          this cipher can be found
          [in this paper](http://www.isg.rhul.ac.uk/~kp/SandPfinal.pdf).
          
          SFTP isn't supported under plan9 until [this
          issue](https://github.com/pkg/sftp/issues/156) is fixed.
          
          Note that since SFTP isn't HTTP based the following flags don't work
          with it: `--dump-headers`, `--dump-bodies`, `--dump-auth`.

This file has been truncated. show original

justme · March 19, 2024, 8:57pm

It does not look to me like these are proper places where to put it. Putting it there will probably disrupt the flow. Perhaps it should be placed where the config "ssh = " and "shell_type = " are explained.

The following paragraph in the documentation page SFTP

Rclone tries to auto-detect what type of shell is used on the server, first time you access the SFTP remote. If a remote shell session is successfully created, it will look for indications that it is CMD or PowerShell, with fall-back to Unix if not something else is detected. If unable to even create a remote shell session, then shell command execution will be disabled entirely. The result is stored in the SFTP remote configuration, in option `shell_type`, so that the auto-detection only have to be performed once. If you manually set a value for this option before first run, the auto-detection will be skipped, and if you set a different value later this will override any existing. Value `none` can be set to avoid any attempts at executing shell commands, e.g. if this is not allowed on the server.

should probably be altered to become:

Rclone tries to auto-detect what type of shell is used on the server, first time you access the SFTP remote. If a remote shell session is successfully created, it will look for indications that it is CMD or PowerShell, with fall-back to Unix if not something else is detected. If unable to even create a remote shell session, then shell command execution will be disabled entirely. The result is stored in the SFTP remote configuration, in option `shell_type`, so that the auto-detection only have to be performed once. If you manually set a value for this option before first run, the auto-detection will be skipped, and if you set a different value later this will override any existing. Value `none` can be set to avoid any attempts at executing shell commands, e.g. if this is not allowed on the server. However, if you have “shell_type = none” in the configuration then you should not have an ssh command such as “ssh = ssh -i path\ssh_id user@ip-address”, you should have “ssh = “ instead.

Please pull or push a request, I am not set up to do that.

kapitainsky · March 21, 2024, 2:37pm

Thx for experimenting and clarification suggestions.

It has been incorporated into docs now.

github.com/rclone/rclone

docs: sftp, clarify `shell_type = none` and `ssh = ` behaviour

rclone:master ← kapitainsky:patch-1

opened 10:56AM - 21 Mar 24 UTC

kapitainsky

+2 -0

Discussed on the forum: https://forum.rclone.org/t/can-rclone-be-made-to-work…-with-an-sftp-server-confining-users-to-an-sftp-jail-and-no-login/44931 #### Checklist - [ x] I have read the [contribution guidelines](https://github.com/rclone/rclone/blob/master/CONTRIBUTING.md#submitting-a-new-feature-or-bug-fix). - [ ] I have added tests for all changes in this PR if appropriate. - [ x] I have added documentation for the changes if appropriate. - [ x] All commit messages are in [house style](https://github.com/rclone/rclone/blob/master/CONTRIBUTING.md#commit-messages). - [ x] I'm done, this Pull Request is ready for review :-)

justme · March 21, 2024, 9:56pm

Looking further down in the same page, there is a paragraph:

The shell type auto-detection logic, described above, means that by default rclone will try to run a shell command the first time a new sftp remote is accessed. If you configure a sftp remote without a config file, e.g. an on the fly remote, rclone will have nowhere to store the result, and it will re-run the command on every access. To avoid this you should explicitly set the shell_type option to the correct value, or to none if you want to prevent rclone from executing any remote shell commands.

Do you think we should add the sentence there too?

kapitainsky · March 22, 2024, 8:48am

IMO no need to repeat the same again. But maybe others think different.

justme · March 22, 2024, 5:25pm

I agree with you; that second paragraph in it's original form without the new sentence is repeating information. Someone may read it and miss our new sentence. Perhaps something should be done to it.

system · April 21, 2024, 5:26pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.

Can rclone be made to work with an SFTP server confining users to an SFTP Jail and no login?

What is the problem you are having with rclone?

Run the command 'rclone version' and share the full output of the command.

Which cloud storage system are you using? (eg Google Drive)

The command you were trying to run (eg rclone copy /tmp remote:tmp)

Please run 'rclone config redacted' and share the full output. If you get

A log from the command that you were trying to run with the -vv flag

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

A log from the command that you were trying to run with the `-vv` flag