SSH Certificate based authentication does not work

Suspected Bug
1

Hi all,

Bug Description

  1. Setup a Client/Host to use Certificate based authentication. The authorized_keys has the appropriate cert-authority entry so that SSH access works fine.

  2. Use the same host/key for rclone sftp backend.

  3. Rclone operations fail --- sample error message for a simple rclone lsd hostname: is below:

2020/09/19 00:20:58 Failed to create file system for "hostname:": NewFs: couldn't connect SSH: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain
  1. Changing the private key to a simple id_rsa keypair works.

Expected Behaviour

The certificate based approach should also work.

Notes

Certificate based authorities means a set up similar to that described below.

The set up is verified working with OpenSSH v7.6 and 8.2 ---- i.e. the one bundled with Ubuntu 18.04LTS and 20.04LTS. The issue

2

The sftp backend doesn't make a callout to an external ssh client, but uses GoLang native sftp library, which in turn uses the native GoLang ssh library.

And it doesn't look like the GoLang ssh library supports cert based auth. I don't see it in https://godoc.org/golang.org/x/crypto/ssh. It is a protocol change... I'm not too surprised; does anything understand certs except openssh?

As the rclone documentation (https://rclone.org/sftp/) says:

SSH Authentication

The SFTP remote supports three authentication methods:

  • Password
  • Key file
  • ssh-agent

Key files should be PEM-encoded private key files. For instance /home/$USER/.ssh/id_rsa . Only unencrypted OpenSSH or PEM encrypted files are supported.

4

What @sweh is saying is rclone can't support it until the under lying GO library supports that rclone uses.

5

There is this in the library docs

I think you should be able to load one and pass it to

Does that look right to you @sweh ? I've never used SSH certificates.

6

I don't think so; I think that "signer" is part of signing the challenge as part of the authentication process. But I'm not too sure.

Going back to basics :slight_smile:

A simple ssh key based authentication goes something like...

  • Client looks at public key
  • Sends key to server (kinda; detail not important)
  • Server says "Hey, I like the look of this. It matches what's in the authorized_keys file. Prove you've got the private key"
  • Complicated math happens
  • Server says "OK, you're good".

The "complicated math" is where signing happens; the client signs a message using the private key, and the server validates it using the public key.

Now one of things hidden, here, is that you don't actually need the pub file to do this. The private key contains a copy of it! We can see this with ssh-keygen -y. For example:

$ ssh-keygen -y -f id_rsa
Enter passphrase: 
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDScBL2j0mwz7swpHBLzE4HjaJ6OpC96sg50aB1LRPbfdojKjiyuc9fC8VSCCCAHdovFihheLsbPyWKGq/tMM4e8JXp0YD2AG63k8FZ98WjAgC5g6UAK+MitmtRxCyjF52JoLioM1R9iACN+8guV0oZ9sSD7DNW2UTZrlgDavrM0QZ17tWu3QEz6U+l1bQwdisZxNpomED5quYI6pZypAdzBKi8HJw7BJDtmoEirPZzTP+T1Iw6pH/7tMDNQ8atk8OAH2VCAYIurXjxXG8+XNoPaqBtU2dF+8gK4TsLU0gqgYRlnE0UIu0aXQb+YSWNV5Xk7oo86F5cMF0WCGCGJ0l9
$ cat id_rsa.pub 
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDScBL2j0mwz7swpHBLzE4HjaJ6OpC96sg50aB1LRPbfdojKjiyuc9fC8VSCCCAHdovFihheLsbPyWKGq/tMM4e8JXp0YD2AG63k8FZ98WjAgC5g6UAK+MitmtRxCyjF52JoLioM1R9iACN+8guV0oZ9sSD7DNW2UTZrlgDavrM0QZ17tWu3QEz6U+l1bQwdisZxNpomED5quYI6pZypAdzBKi8HJw7BJDtmoEirPZzTP+T1Iw6pH/7tMDNQ8atk8OAH2VCAYIurXjxXG8+XNoPaqBtU2dF+8gK4TsLU0gqgYRlnE0UIu0aXQb+YSWNV5Xk7oo86F5cMF0WCGCGJ0l9 sweh@test1.spuddy.org

You can see the key is the same, just missing the comment at the end.

So why, if the public key is in the private key file do we need a .pub file? Well, if you looked carefully I needed to enter the passphrase to unlock the encrypted private key file. By having a separate public key file it means we can send the public key to the server and then only need to unlock the one(s) being used to authenticate. This gets important if there's a lot of possible keys (why unlock those that can't be used, anyway?)

So let's unencrypt the private key (to make this easier, going forward) with ssh-keygen -p and add it to the authorized_keys file.

Now we can see an example of OpenSSH and the files it opens:

$ strace -ff ssh test1.spuddy.org uname -a 2>&1 | grep ^open.*id_rsa
open("/home/sweh/.ssh/id_rsa", O_RDONLY) = 4
open("/home/sweh/.ssh/id_rsa", O_RDONLY) = 4
open("/home/sweh/.ssh/id_rsa", O_RDONLY) = 4
open("/home/sweh/.ssh/id_rsa.pub", O_RDONLY) = 4
open("/home/sweh/.ssh/id_rsa-cert", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/home/sweh/.ssh/id_rsa-cert.pub", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/home/sweh/.ssh/id_rsa", O_RDONLY) = 4
$

Let's create a simple rclone test config:

[sftptest]
type = sftp
host = test1.spuddy.org
user = sweh
key_file = /home/sweh/.ssh/id_rsa
use_insecure_cipher = false

And what do we see with this one?

$ strace -ff rclone lsd sftptest: 2>&1 | grep id_rsa
[pid  4957] openat(AT_FDCWD, "/home/sweh/.ssh/id_rsa", O_RDONLY|O_CLOEXEC) = 8
$

It's not opening the pub file at all, so it must be extracting the public key from the private key.

However, with a signed certificate the public key file is different. It contains the original information plus a tonne of extra data. This isn't in the private key file.

So what might work (I don't know enough about the GoLang ssh/sftp libraries to be sure this will work) would be to allow the user to optionally specify the public key as well as the private key; if the user specifies the cert signed public key then we might get lucky!

1 Like
7

But all the examples I can see also include just passing the unlocked private key and letting it extract the public key :frowning:

8

I was thinking that if you can make a signer from a certificate then that's what you need for SSH Auth. See the signers being added here.

So I think you call this func with the cert and the signer for the private key so we'd probably call this after making the signers we make already.

I found this issue

Which explains how to parse a certificate and it looks like you are right, you can use it to parse public certs too.

Seems plausible to me! Fancy giving it a go?

9

So I think I have sample code that shows sftp being used with certs. You end up having to use ParseAuthorizedKeys 'cos ParsePublicKeys wants wire-format, not disk format.

package main

import (
	"github.com/pkg/sftp"
	"golang.org/x/crypto/ssh"
	"io/ioutil"
	"log"
)

func main() {
	key, err := ioutil.ReadFile("id_rsa")
	if err != nil {
		log.Fatalf("unable to read private key: %v", err)
	}

	// Create the Signer for this private key.
	priv_signer, err := ssh.ParsePrivateKey(key)
	if err != nil {
		log.Fatalf("unable to parse private key: %v", err)
	}

	certfile, err := ioutil.ReadFile("id_rsa-cert.pub")
	if err != nil {
		log.Fatalf("unable to read cert file: %v", err)
	}

	// Yes, this is AUTHORIZED key.. cos it's on-disk format
	pk, _, _, _, err := ssh.ParseAuthorizedKey(certfile)
	if err != nil {
		log.Fatalf("unable to parse cert file: %v", err)
	}

	// And the signer for this, which includes the private key signer
	// This is what we'll pass to the ssh client.
	certSigner, err := ssh.NewCertSigner(pk.(*ssh.Certificate), priv_signer)
	if err != nil {
		log.Fatalf("error generating cert signer: %v", err)
	}

	config := &ssh.ClientConfig{
		User: "sweh",
		Auth: []ssh.AuthMethod{
			// Use the PublicKeys method for remote authentication.
			ssh.PublicKeys(certSigner),
		},
		HostKeyCallback: ssh.InsecureIgnoreHostKey(),
	}

	addr := "test1.spuddy.org:22"

	conn, err := ssh.Dial("tcp", addr, config)
	if err != nil {
		panic("Failed to dial: " + err.Error())
	}
	client, err := sftp.NewClient(conn)
	if err != nil {
		panic("Failed to create client: " + err.Error())
	}
	// Close connection
	defer client.Close()
	cwd, err := client.Getwd()
	println("Current working directory:", cwd)
}

If I run this I get

Current working directory: /home/sweh

And the server side logs show:

Sep 19 18:13:33 test1 sshd[10883]: Accepted publickey for sweh from 10.0.0.137 port 50410 ssh2: RSA-CERT ID user_sweh (serial 0) CA RSA SHA256:9Uu2dQc6N4hgwv5Yz/UVU+fJfUTJzShQ0i4LdU7owEU
Sep 19 18:13:33 test1 sshd[10883]: pam_unix(sshd:session): session opened for user sweh by (uid=0)
Sep 19 18:13:33 test1 sshd[10883]: pam_unix(sshd:session): session closed for user sweh

I think should be enough to allow you to modify the sftp backend to have another optional value!

10

That looks really promising :smiley:

So does the certificate use the standard private key, or does it normally have its own? The strace above makes me think that it should have its own.

I think you could add a certificate private key to into an an ssh agent so I think if the certificate option is set then you'd want to wrap that in the certificate wouldn't you?

11

It's the standard private key; that's not changed 'cos it key material is still the same; it's just the public key has a CA-signed certificate added to it.

In a general scenario you end up with files like id_rsa, id_rsa.pub and id_rsa-cert.pub. There's no id_rsa-cert, although OpenSSH will look for one as a belts-and-suspenders.

That's why my example code uses id_rsa and id_rsa-cert.pub as the two files needed.

I think the OpenSSH Agent code was modified to handle certs by adding the cert public key as well as the private key

$ ssh-add -l  
The agent has no identities.
$ ssh-add
Identity added: /home/sweh/.ssh/id_rsa (/home/sweh/.ssh/id_rsa)
Certificate added: /home/sweh/.ssh/id_rsa-cert.pub (user_sweh)
12

As a quick note - it's also possible to concatenate the certificate to the regular id_rsa and the singular combined file will also be accepted by sshd.

13

FWIW, I could probably code up the ability for this to work, but I have no idea how to create a test suite for it. And I know Nick is really hot on test suites :slight_smile:

If Nick is willing to accept a PR without tests then I think I could code one up.

14

The quick guide in the OP shows how to set up the certificates but I can also assist in testing on my setup if it helps?

15

Ah, no; I can setup cert based auth (I documented this 4 years ago at https://www.sweharris.org/post/2016-10-30-ssh-certs/ ) and have a test rig that I used earlier in creating the simple GoLang test code. I can demonstrate my code works :slight_smile:

The testing I'm talking about are the internal test routines. Although looking at the tests in https://github.com/rclone/rclone/tree/master/backend/sftp it's not clear they're doing anything "real", anyway! Probably for the same reason!

16

That would be great!

The backend test suites test the functionality of the backend quite thouroughly. However things like different ways of logging in are very hard to write tests for. What can be done is to put it in the integration test suite which involves setting up a remote and running it against a server (probably something in a docker container).

We don't have tests for the other ways of ssh auth though so I think I'd be happy to take it without tests. If it breaks a lot then we can set up an integration test for it.

FYI

The integration tests are defined here

And here is an example of the docker startup routines

There is also a config entry which isn't checked in.

17

Looking good :slight_smile:

$ cat /tmp/TSTKEYS/conf 
[sftpcert]
type = sftp
host = test1.spuddy.org
key_file = /tmp/TSTKEYS/id_rsa
pubkey_file = /tmp/TSTKEYS/id_rsa-cert.pub

$ ./rclone --config /tmp/TSTKEYS/conf ls sftpcert:/tmp/XX 
      131 README
  1716863 hosts
   594617 justdomains

And on the server:

Sep 24 14:23:59 test1 sshd[3391]: Accepted publickey for sweh from 10.0.0.137:4462 port 44108 ssh2: RSA-CERT ID user_sweh (serial 0) CA RSA SHA256:9Uu2dQc6N4hgwv5Yz/UVU+fJfUTJzShQ0i4LdU7owEU
Sep 24 14:23:59 test1 sshd[3391]: pam_unix(sshd:session): session opened for user sweh by (uid=0)
Sep 24 14:23:59 test1 sshd[3391]: pam_unix(sshd:session): session closed for user sweh
18

With any luck, https://github.com/rclone/rclone/pull/4625 :slight_smile:

19

Does the PR work if the private key and cert are concatenated into one file and only the key_file option (OpenSSH allows this)

20

You specify the same file in both places.

eg

$ cat id_rsa-cert.pub id_rsa > tstkey

$ chmod 600 tstkey

$ # verify it works
$ ssh -i tstkey test1 hostname
test1.spuddy.org

$ cat conf
[sftpcert]
type = sftp
host = test1.spuddy.org
key_file = /tmp/TSTKEYS/tstkey
pubkey_file = /tmp/TSTKEYS/tstkey

$ ./rclone --config /tmp/TSTKEYS/conf ls sftpcert:/tmp/XX
      131 README
  1716863 hosts
   594617 justdomains
2 Likes
21

That looks great.

No issue manually editing the rclone.conf but curious will setting this using rclone config be possible?