Mount Ubuntu homeserver on startup with rclone trough sftp on linux

[Medeon]

What is the problem you are having with rclone?

I can mount my ubuntu homeserver trough sftp using the rclone mount command. What also works is:

mount raspi: /home/ejan/Openbaar/Homeserver -t rclone -o config=/home/ejan/.config/rclone/rclone.conf,vfs_cache_mode=writes,sftp_key_file=/etc/ssh/id_rclone

Unfortunately I can't mount using a systemd script on startup, however when I start the systemd mount service after booting up my computer mounting my Ubuntu server's homefolder works.

Run the command 'rclone version' and share the full output of the command.

rclone v1.63.0

• os/version: arch "rolling" (64 bit)
• os/kernel: 6.1.38-2-lts (x86_64)
• os/type: linux
• os/arch: amd64
• go/version: go1.20.5
• go/linking: dynamic
• go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Ubuntu homeserver on Raspberrypi trough SFTP.

The command you were trying to run (eg rclone copy /tmp remote:tmp)

[Unit]
Description=rclone: Remote FUSE filesystem for raspi Ubuntu homeserver
Documentation=man:rclone(1)
#After=network-online.target
#Wants=network-online.target
AssertPathIsDirectory=/home/ejan/Openbaar/Homeserver

[Service]
Type=notify
Environment="SSH_AUTH_SOCK=/run/user/1000/keyring/ssh"
ExecStart= \
        /usr/bin/rclone mount \
        --config=/home/ejan/.config/rclone/rclone.conf \
        --sftp-key-file=/etc/ssh/id_rclone \
	--sftp-pubkey-file=/etc/ssh/id_rclone.pub.pem \
	--cache-dir=/var/rclone \
	--vfs-cache-mode writes \
        --log-level INFO \
        --log-file /tmp/rclone-raspi.log \
        --umask 022 \
	--allow-other \
        raspi: /home/ejan/Openbaar/Homeserver
ExecStop=/bin/fusermount -u /home/ejan/Openbaar/Homeserver

[Install]
WantedBy=default.target

The rclone config contents with secrets removed.

[raspi]
type = sftp
host = 192.168.2.242
key_file = /etc/ssh/id_rclone
key_file_pass = My_Super_Secret_Key_File_Pass
pubkey_file = /etc/ssh/id_rclone.pub.pem
key_use_agent = true
use_insecure_cipher = false
idle_timeout = 10m0s
ciphers = aes128-ctr aes192-ctr aes256-ctr aes128-gcm@openssh.com aes256-gcm@openssh.com
key_exchange = diffie-hellman-group-exchange-sha256 curve25519-sha256
host_key_algorithms = ssh-rsa ssh-ed25519
shell_type = unix
md5sum_command = md5sum
sha1sum_command = sha1sum

A log from the command with the -vv flag

cat /tmp/rclone-raspi.log
2023/07/29 00:11:29 Failed to create file system for "raspi:": couldn't connect to ssh-agent: Error connecting to SSH_AUTH_SOCK: dial unix /run/user/1000/keyring/ssh: connect: no such file or directory

[kapitainsky]

It is not really rclone problem...

You have to investigate on your OS how to start ssh-agent before mount systemd service.

Or actually think if you really need ssh-agent when you provide key file explicitly

[Medeon]

Actually, this is what's so confusing to me: If I go without the option key_use_agent I get this feedback after rclone mount raspi: /home/ejan/Openbaar/Homeserver

2023/07/29 16:29:45 Failed to create file system for "raspi:": failed to parse private key file: ssh: not an encrypted key

However this is what the official rclone documentation states on creating a PEM-encoded ssh keyfile for sftp:

Key files should be PEM-encoded private key files. For instance /home/$USER/.ssh/id_rsa . Only unencrypted OpenSSH or PEM encrypted files are supported.

[kapitainsky]

How did you create your key? Does it have password

[Medeon]

Like this: ssh-keygen -P "" -a 100 -t rsa -b 4096 -m pem -f id_rclone -C "ejan@rclonemount" I did not set a passphrase. Since this created just a private and public key, I followed some instructions on how to convert your public key to a pem encoded key (not sure if this is the right way) with: ssh-keygen -f id_rclone.pub -m 'PEM' -e > id_rclone.pub.pem

[kapitainsky]

I think you over engineered this a bit. There is no need to use every single flag from documentation.

Firstly as per docs:

If you have a certificate you may use it to sign your public key, creating a separate SSH user certificate that should be used instead of the plain public key extracted from the private key. Then you must provide the path to the user certificate public key file in pubkey_file.

Note: This is not the traditional public key paired with your private key, typically saved as /home/$USER/.ssh/id_rsa.pub. Setting this path in pubkey_file will not work.

Based on your description how you generate it I think in your case just remove pubkey_file

then

rclone mount raspi-test: /home/ejan/Openbaar/Homeserver

should work with:

[raspi-test]
type = sftp
host = 192.168.2.242
user = ???
key_file = /etc/ssh/id_rclone

It will also work in systemd during startup. If you want to use ssh-agent you have to make sure that it is running before.

@Medeon

Actually, this is what's so confusing to me: If I go without the option key_use_agent I get this feedback after rclone mount raspi: /home/ejan/Openbaar/Homeserver

2023/07/29 16:29:45 Failed to create file system for "raspi:": failed to parse private key file: ssh: not an encrypted key

I suspect that when you do not use agent rclone tries to decrypt key_file using key_file_pass (which I have no idea why you use in your config - remove it too). Your private key is not encrypted so it fails.

[Medeon]

Thank you very much, that indeed works fine. I guess the confusing part for me is whether or not to use an encrypted ssh key and because of being restricted to use a PEM encoded public keyfile. So correct me if I'm wrong, you should only use a pub_keyfile if you set a passphrase when you generate a ssh key? You then should create a pem encoded public keyfile?

[kapitainsky]

I agree that documentation is not 100% clear. And it is not only you getting confused:

docs/sftp: more detailed explanation of pubkey file and certificate

I am not sure myself exact use of pub_keyfile - would have to investigate:)

[kapitainsky]

There is some explanation in this thread:

SSH Certificate based authentication does not work

It's the standard private key; that's not changed 'cos it key material is still the same; it's just the public key has a CA-signed certificate added to it. In a general scenario you end up with files like id_rsa, id_rsa.pub and id_rsa-cert.pub. There's no id_rsa-cert, although OpenSSH will look for one as a belts-and-suspenders. That's why my example code uses id_rsa and id_rsa-cert.pub as the two files needed. I think the OpenSSH Agent code was modified to handle certs by adding the cert pu…

[kapitainsky]

@Medeon

Thank you very much, that indeed works fine. I guess the confusing part for me is whether or not to use an encrypted ssh key and because of being restricted to use a PEM encoded public keyfile. So correct me if I'm wrong, you should only use a pub_keyfile if you set a passphrase when you generate a ssh key? You then should create a pem encoded public keyfile?

Ok. I think I understand it now. pub_keyfile has nothing to do with encryption and passphrases...

You use encrypted private key (there is no need to encrypt public one) and --sftp-key-file-pass flag pointing to password file when your goal is to protect your key from unauthorized use. It is up to you how you solve password protection part. Usually this is what ssh-agents do. But with --sftp-key-file-pass you can have your own solution.

Now all your ssh infrastructure can be designed to only trust signed keys. If you have certificate you can use it to sign keys to be trusted (assumption is all machines know and trust this certificate). In such case you can use pub_keyfile pointing to signed public key (not encrypted) to explicitly advertise its trustworthiness. And thx to signing it can be verified by others.

You can use both/one/none from the above. When using all it requires some openssh standard reading as things quickly are becoming a bit complex.

[sweh (Stephen Harris)]

FWIW, if you want to understand cert based authn I have a couple of blog posts at Using SSH certificates · Ramblings of a Unix Geek and Using SSH certificates - revisited · Ramblings of a Unix Geek which describes how they work with OpenSSH.

The rclone implementation most "just works" by using pub_keyfile to point to the certificate.

[Medeon]

Thanks again for taking a deep dive into this subject. I really appreciate your efforts. It seems like option 1 looks a lot like the way full disk luks encryption works. I'll have to look into this further since I do care about security.

[kapitainsky]

@Medeon

It seems like option 1 looks a lot like the way full disk luks encryption works

Agree - the same concept. Data is encrypted until keys/password to the kingdom is provided:) Hard to implement for unattended server reboot though.