Sftp server_command help

bjornsmith · August 31, 2023, 3:45pm

What is the problem you are having with rclone?

I have files on the server that is only readable by either root or a specific user. Rclone cannot sync these simply because the permissions does not allow it.
I would like to use sudo or similar when launching the sftp server server side.

I found the server_command - and I have used that but its not working at all.
When I run the command manually:

sftp -s "sudo /usr/libexec/openssh/sftp-server" bbs@cephnas.root.dom:/mnt/storage

It works just fine, meaning that I get a sftp connection.

So either I am doing something wrong and the arguments for server_command is not as I expect - or server_command is not working.

It would be great if I either can get an example of a working server_command or something else.

I looked at the commit where support was added, but the commit unfortunately did not include any tests or examples.

Run the command 'rclone version' and share the full output of the command.

rclone v1.56.1

os/version: debian 11.5 (64 bit)
os/kernel: 5.15.107+truenas (x86_64)
os/type: linux
os/arch: amd64
go/version: go1.16.8
go/linking: static
go/tags: none

Which cloud storage system are you using? (eg Google Drive)

None - I am trying to rclone a filesystem from a remote server to the local server

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

/usr/bin/rclone sync --progress --stats-one-line --create-empty-src-dirs --stats=30s --transfers 4 --checkers 4 --config /home/admin/.config/rclone/rclone.conf cephnas:/mnt/storage /mnt/tank/backup

The rclone config contents with secrets removed.

[cephnas]
type = sftp
server_command =  "sudo /usr/libexec/openssh/sftp-server"
host = cephnas.root.dom
user = bbs
use_insecure_cipher = true
key_file = /home/admin/.ssh/id_rsa
md5sum_command = md5 -r
sha1sum_command = sha1 -r

A log from the command with the `-vv` flag

2023/08/31 18:40:09 DEBUG : rclone: Version "v1.56.1" starting with parameters ["/usr/bin/rclone" "sync" "-vv" "--progress" "--stats-one-line" "--create-empty-src-dirs" "--stats=30s" "--transfers" "4" "--checkers" "4" "--config" "/home/admin/.config/rclone/rclone.conf" "cephnas:/mnt/storage" "/mnt/tank/backup"]
2023/08/31 18:40:09 DEBUG : Creating backend with remote "cephnas:/mnt/storage"
2023/08/31 18:40:09 DEBUG : Using config file from "/home/admin/.config/rclone/rclone.conf"
2023/08/31 18:40:09 DEBUG : sftp://bbs@cephnas.root.dom:22//mnt/storage: New connection 192.168.0.200:49476->192.168.2.200:22 to "SSH-2.0-OpenSSH_8.0"
2023/08/31 18:40:09 DEBUG : pacer: low level retry 1/10 (error couldn't initialise SFTP: EOF)
2023/08/31 18:40:09 DEBUG : pacer: Rate limited, increasing sleep to 200ms
2023/08/31 18:40:09 DEBUG : sftp://bbs@cephnas.root.dom:22//mnt/storage: New connection 192.168.0.200:49488->192.168.2.200:22 to "SSH-2.0-OpenSSH_8.0"
2023/08/31 18:40:09 DEBUG : pacer: low level retry 2/10 (error couldn't initialise SFTP: EOF)
2023/08/31 18:40:09 DEBUG : pacer: Rate limited, increasing sleep to 400ms
2023/08/31 18:40:09 DEBUG : sftp://bbs@cephnas.root.dom:22//mnt/storage: New connection 192.168.0.200:49498->192.168.2.200:22 to "SSH-2.0-OpenSSH_8.0"
2023/08/31 18:40:09 DEBUG : pacer: low level retry 3/10 (error couldn't initialise SFTP: EOF)
2023/08/31 18:40:09 DEBUG : pacer: Rate limited, increasing sleep to 800ms
2023/08/31 18:40:10 DEBUG : sftp://bbs@cephnas.root.dom:22//mnt/storage: New connection 192.168.0.200:49504->192.168.2.200:22 to "SSH-2.0-OpenSSH_8.0"
2023/08/31 18:40:10 DEBUG : pacer: low level retry 4/10 (error couldn't initialise SFTP: EOF)
2023/08/31 18:40:10 DEBUG : pacer: Rate limited, increasing sleep to 1.6s
2023/08/31 18:40:11 DEBUG : sftp://bbs@cephnas.root.dom:22//mnt/storage: New connection 192.168.0.200:49516->192.168.2.200:22 to "SSH-2.0-OpenSSH_8.0"
2023/08/31 18:40:11 DEBUG : pacer: low level retry 5/10 (error couldn't initialise SFTP: EOF)
2023/08/31 18:40:11 DEBUG : pacer: Rate limited, increasing sleep to 2s
2023/08/31 18:40:12 DEBUG : sftp://bbs@cephnas.root.dom:22//mnt/storage: New connection 192.168.0.200:49528->192.168.2.200:22 to "SSH-2.0-OpenSSH_8.0"
2023/08/31 18:40:12 DEBUG : pacer: low level retry 6/10 (error couldn't initialise SFTP: EOF)
2023/08/31 18:40:14 DEBUG : sftp://bbs@cephnas.root.dom:22//mnt/storage: New connection 192.168.0.200:49544->192.168.2.200:22 to "SSH-2.0-OpenSSH_8.0"
2023/08/31 18:40:14 DEBUG : pacer: low level retry 7/10 (error couldn't initialise SFTP: EOF)
2023/08/31 18:40:16 DEBUG : sftp://bbs@cephnas.root.dom:22//mnt/storage: New connection 192.168.0.200:43026->192.168.2.200:22 to "SSH-2.0-OpenSSH_8.0"
2023/08/31 18:40:16 DEBUG : pacer: low level retry 8/10 (error couldn't initialise SFTP: EOF)
2023/08/31 18:40:18 DEBUG : sftp://bbs@cephnas.root.dom:22//mnt/storage: New connection 192.168.0.200:43038->192.168.2.200:22 to "SSH-2.0-OpenSSH_8.0"
2023/08/31 18:40:18 DEBUG : pacer: low level retry 9/10 (error couldn't initialise SFTP: EOF)
2023/08/31 18:40:20 DEBUG : sftp://bbs@cephnas.root.dom:22//mnt/storage: New connection 192.168.0.200:43050->192.168.2.200:22 to "SSH-2.0-OpenSSH_8.0"
2023/08/31 18:40:20 DEBUG : pacer: low level retry 10/10 (error couldn't initialise SFTP: EOF)
2023/08/31 18:40:20 Failed to create file system for "cephnas:/mnt/storage": NewFs: couldn't initialise SFTP: EOF

kapitainsky · August 31, 2023, 3:51pm

Update your rclone to the latest version first. It is waste of everybody's time to investigate issues in some old releases.

bjornsmith · August 31, 2023, 3:57pm

I got it working.
server_command should not be i quotes.

So a working config for me is:

[cephnas]
type = sftp
server_command = sudo /usr/libexec/openssh/sftp-server
host = cephnas.root.dom
user = bbs
use_insecure_cipher = true
key_file = /home/admin/.ssh/id_rsa
md5sum_command = md5 -r
sha1sum_command = sha1 -r

I think it would be awesome if the documentation was improved with an example - and also a note that commands should not be enclosed in qoutes.

Animosity022 · August 31, 2023, 4:26pm

It would be awesome if you found that to be awesome enough to submit a PR to update it

bjornsmith · August 31, 2023, 4:27pm

I can do that - is the docs also on github?

Animosity022 · August 31, 2023, 4:29pm

Yes sir!

github.com

rclone/rclone/blob/master/docs/content/sftp.md

---
title: "SFTP"
description: "SFTP"
versionIntroduced: "v1.36"
---

# {{< icon "fa fa-server" >}} SFTP

SFTP is the [Secure (or SSH) File Transfer
Protocol](https://en.wikipedia.org/wiki/SSH_File_Transfer_Protocol).

The SFTP backend can be used with a number of different providers:

{{< provider_list >}}
{{< provider name="Hetzner Storage Box" home="https://www.hetzner.com/storage/storage-box" config="/sftp/#hetzner-storage-box">}}
{{< provider name="rsync.net" home="https://rsync.net/products/rclone.html" config="/sftp/#rsync-net">}}
{{< /provider_list >}}

SFTP runs over SSH v2 and is installed as standard with most modern
SSH installations.

This file has been truncated. show original

Click on the pencil and it'll work through a PR request.

bjornsmith · August 31, 2023, 4:39pm

There you go - a pull request is created.

kapitainsky · August 31, 2023, 4:43pm

If you want to add example to the --sftp-server-command flag description

--sftp-server-command

Specifies the path or command to run a sftp server on the remote host.

The subsystem option is ignored when server_command is defined.

Properties:
Config: server_command
Env Var: RCLONE_SFTP_SERVER_COMMAND
Type: string
Required: false

please note that this part of help is auto generated (you can see flags section is enclosed in {{< rem autogenerated options start" - DO NOT EDIT - instead edit fs.RegInfo in backend/sftp/sftp.go then run make backenddocs" >}}) from source in

github.com

rclone/rclone/blob/master/backend/sftp/sftp.go

//go:build !plan9

// Package sftp provides a filesystem interface using github.com/pkg/sftp
package sftp

import (
	"bytes"
	"context"
	"errors"
	"fmt"
	"io"
	iofs "io/fs"
	"os"
	"path"
	"regexp"
	"strconv"
	"strings"
	"sync"
	"sync/atomic"
	"time"

This file has been truncated. show original

kapitainsky · August 31, 2023, 4:45pm

Thank you for PR but sorry I was too slow with detailed explanation:) Your change should go to another file.

bjornsmith · August 31, 2023, 4:48pm

I see - so I did it the entirely wrong way.

I do not have go installed anywhere - so I cannot just run make backenddocs - or is there any other way I can contribute this extra docs without actually having to install go on a system somewhere?

Would it be acceptable if I added a section below - similar to the Limitations section?

kapitainsky · August 31, 2023, 4:50pm

You do not have to run anything. This is only comment how to generate html file (it happens during release).

Your edit should go to sftp.go file - you will see how it is formatted when you edit it.

bjornsmith · August 31, 2023, 4:50pm

okay - I will do that then - sorry for the confusion

bjornsmith · August 31, 2023, 4:58pm

github.com/rclone/rclone

Add note regarding format of server_command

rclone:master ← bjornbouetsmith:patch-1

opened 04:58PM - 31 Aug 23 UTC

bjornbouetsmith

+10 -1

Elaborate exactly how server_command should be used in the configuration file …  #### What is the purpose of this change? Help people understand the exact format of server_command in the configuration file  #### Was the change discussed in an issue or in the forum before?  https://forum.rclone.org/t/sftp-server-command-help/41412 #### Checklist - [x] I have read the [contribution guidelines](https://github.com/rclone/rclone/blob/master/CONTRIBUTING.md#submitting-a-new-feature-or-bug-fix). - [x] I have added tests for all changes in this PR if appropriate. - [x] I have added documentation for the changes if appropriate. - [x] All commit messages are in [house style](https://github.com/rclone/rclone/blob/master/CONTRIBUTING.md#commit-messages). - [x] I'm done, this Pull Request is ready for review :-)

I hope thats better - I was unsure how to embed markdown format inside the documentation.

kapitainsky · August 31, 2023, 4:59pm

Cool - now looks perfect:)

bjornsmith · August 31, 2023, 5:02pm

Is there someone I can ping - to get the PR approved and merged?

kapitainsky · August 31, 2023, 5:06pm

No rush:) Even when merged it wont appear in docs until the next version release. But you can see merged beta docs here - https://tip.rclone.org/

all new PRs are reviewed pretty much daily and processed approved or commented.

bjornsmith · August 31, 2023, 5:15pm

Cool - thanks

Is it true that if I prefix a line with 4 spaces, then it turns into markdown?

I took at look at one of the other examples - and that seem to be the case

kapitainsky · August 31, 2023, 5:16pm

Frankly I am not sure myself:) But I would do exactly what you said - check other flags:)

kapitainsky · August 31, 2023, 5:17pm

I think it is 4 spaces?

bjornsmith · August 31, 2023, 5:20pm

Thats also what I added - I copied the space from one of the other examples I just thought it was 3 - but I updated my comment above.

That should also be added to the "contributing" documentation - how to include markdown as part of the automatic generated docomentation