Rclone fails ssh handshakes with rsync.net's SFTP when a known hosts file is specified

What is the problem you are having with rclone?

rclone's ssh implementation used for SFTP always rejects the signatures of rsync.net's host keys when a known_hosts file is provided.

couldn't connect SSH: ssh: handshake failed: knownhosts: key mismatch

Plain old ssh and sftp don't encounter the error, and I also tested SFTP on Windows Filezilla with no problems (which all required me to verify the host's key).

Run the command 'rclone version' and share the full output of the command.

rclone v1.57.0
- os/version: Microsoft Windows 10 Enterprise 2009 (64 bit)
- os/kernel: 10.0.19043.1526 (x86_64)
- os/type: windows
- os/arch: amd64
- go/version: go1.17.2
- go/linking: dynamic
- go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

Rsync.net's SFTP interface.

The command you were trying to run (eg rclone copy /tmp remote:tmp)

This sample command uses a dummy user ID, which works to replicate the error because it happens even before rclone can prompt for a password. I've tested it with my actual user ID and with --sftp-key-file instead, and both encounter the same error.

I also tested the same command on an Alpine Linux VM and can get the same signature error.

rclone \
  -vv \
  --config=./rsyncnet.config \
  --sftp-host=hk-s020.rsync.net \
  --sftp-user=12345 \
  --sftp-ask-password=true \
  --sftp-known-hosts-file=./rsyncnet_known_hosts \
  ls rsyncnet:.

The rclone config contents with secrets removed.

[rsyncnet]
type = sftp

And here's the rsyncnet_known_hosts file. I also tried a few random servers from https://www.rsync.net/resources/fingerprints.txt and got the same error. Although one of the random ones did have the error from this old thread

hk-s020.rsync.net ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAILcPl9x9JfRFwsn09NnDw/xBZbAN80ZQck+h6AqlVqPH

A log from the command with the -vv flag

2022/02/11 06:37:11 DEBUG : rclone: Version "v1.57.0" starting with parameters ["C:\\REDACTED_PATH\\rclone.exe" "-vv" "--config=./rsyncnet.config" "--sftp-host=hk-s020.rsync.net" "--sftp-user=12345" "--sftp-ask-password=true" "--sftp-known-hosts-file=./rsyncnet_known_hosts" "ls" "rsyncnet:."]
2022/02/11 06:37:11 DEBUG : Creating backend with remote "rsyncnet:."
2022/02/11 06:37:11 DEBUG : Using config file from "C:\\REDACTED_PATH\\rsyncnet.config"
2022/02/11 06:37:12 DEBUG : rsyncnet: detected overridden config - adding "{-MFU0}" suffix to name
2022/02/11 06:37:13 DEBUG : pacer: low level retry 1/10 (error couldn't connect SSH: ssh: handshake failed: knownhosts: key mismatch)
2022/02/11 06:37:13 DEBUG : pacer: Rate limited, increasing sleep to 200ms
2022/02/11 06:37:15 DEBUG : pacer: low level retry 2/10 (error couldn't connect SSH: ssh: handshake failed: knownhosts: key mismatch)
2022/02/11 06:37:15 DEBUG : pacer: Rate limited, increasing sleep to 400ms
2022/02/11 06:37:16 DEBUG : pacer: low level retry 3/10 (error couldn't connect SSH: ssh: handshake failed: knownhosts: key mismatch)
2022/02/11 06:37:16 DEBUG : pacer: Rate limited, increasing sleep to 800ms
2022/02/11 06:37:17 DEBUG : pacer: low level retry 4/10 (error couldn't connect SSH: ssh: handshake failed: knownhosts: key mismatch)
2022/02/11 06:37:17 DEBUG : pacer: Rate limited, increasing sleep to 1.6s
2022/02/11 06:37:18 DEBUG : pacer: low level retry 5/10 (error couldn't connect SSH: ssh: handshake failed: knownhosts: key mismatch)
2022/02/11 06:37:18 DEBUG : pacer: Rate limited, increasing sleep to 2s
2022/02/11 06:37:20 DEBUG : pacer: low level retry 6/10 (error couldn't connect SSH: ssh: handshake failed: knownhosts: key mismatch)
2022/02/11 06:37:22 DEBUG : pacer: low level retry 7/10 (error couldn't connect SSH: ssh: handshake failed: knownhosts: key mismatch)
2022/02/11 06:37:24 DEBUG : pacer: low level retry 8/10 (error couldn't connect SSH: ssh: handshake failed: knownhosts: key mismatch)
2022/02/11 06:37:26 DEBUG : pacer: low level retry 9/10 (error couldn't connect SSH: ssh: handshake failed: knownhosts: key mismatch)
2022/02/11 06:37:28 DEBUG : pacer: low level retry 10/10 (error couldn't connect SSH: ssh: handshake failed: knownhosts: key mismatch)
2022/02/11 06:37:28 Failed to create file system for "rsyncnet:.": NewFs: couldn't connect SSH: ssh: handshake failed: knownhosts: key mismatch

FWIW, I just did a "ssh rsync.net" and accepted the key it supplied.

This was

rsync.net ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKFjGEbhu9Spooi7N2N7IjoS67gaA0EFAV8xpEaxvNX13uorFjECq3dCL6UxwV/JXdr8A4LNcBrC4v0z88DSwp8=

With that in my known hosts file the connection succeeds (but, obviously fails to authenticate since I don't have an account).

rclone -vv --config=./x --sftp-host=rsync.net --sftp-known-hosts-file=y ls rsync:
2022/02/10 18:41:06 DEBUG : rclone: Version "v1.56.0" starting with parameters ["rclone" "-vv" "--config=./x" "--sftp-host=rsync.net" "--sftp-known-hosts-file=y" "ls" "rsync:"]
2022/02/10 18:41:06 DEBUG : Creating backend with remote "rsync:"
2022/02/10 18:41:06 DEBUG : Using config file from "/home/sweh/x"
2022/02/10 18:41:06 DEBUG : rsync: detected overridden config - adding "{FwXzO}" suffix to name
2022/02/10 18:41:06 DEBUG : pacer: low level retry 1/10 (error couldn't connect SSH: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain)
2022/02/10 18:41:06 DEBUG : pacer: Rate limited, increasing sleep to 200ms
^C

% cat x
[rsync]
type = sftp

% cat y
rsync.net ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKFjGEbhu9Spooi7N2N7IjoS67gaA0EFAV8xpEaxvNX13uorFjECq3dCL6UxwV/JXdr8A4LNcBrC4v0z88DSwp8=
1 Like

hi,
hopefully not too off-topic but you do not stop by much these days.

i have a rsync.net account, works well.
having been meaning to use it but when i try, does not work, mabye you have an idea.

this is the contents of the known_hosts_file
usw-s003.rsync.net ssh-ed25519 jJCISHUXooV9/rh7kSeqiSuPvOVWBFCOe2CNkwTO46o

and this is the output i get this

rclone lsd rsync: -vv 
DEBUG : rclone: Version "v1.57.0" starting with parameters ["c:\\data\\rclone\\scripts\\rclone.exe" "lsd" "rsync:" "-vv"]
DEBUG : Creating backend with remote "rsync:"
Failed to create file system for "rsync:": couldn't parse known_hosts_file: knownhosts: C:\data\c\BitviseSSHClient\rsync\known_hosts:1: illegal base64 data at input byte 40
[rsync]
type = sftp
host = usw-s003.rsync.net
user = 0000
key_file = C:\data\c\BitviseSSHClient\rsync\rync.priv
known_hosts_file = C:\data\c\BitviseSSHClient\rsync\known_hosts
md5sum_command = md5 -r
sha1sum_command = sha1 -r

Your known-hosts entry is corrupt. It looks too short, and isn't valid base64.

Fix that :slight_smile:

I think it should read

usw-s003.rsync.net ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIJ1WLJ+V5w1u0SIT9mP4KccSOZyj9Q380IWXKIjOLz9T
1 Like

FWIW, I just did a "ssh rsync.net" and accepted the key it supplied.

Oh, tried it out and I see a discrepancy on my end.

When I ssh rsync.net, my ssh picks up the ssh-ed25519 key, while yours is getting the ecdsa-sha2-nistp256 key. An ssh-keyscan rsync.net shows that it has 3 keys:

rsync.net ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvu7xQIf1RxvVEfJcEAumXMd7L4ncwc2uxOYA94Y7tjbiq1sIJPP4iUCgm8mnG0Q//H/JHHMBoNzClYQX1CFPW3vmiDWTH8JIHN0Aitv+HeKPsHCZsBpG48VmsydSGomg8CoJ7F8PfHnBjov0p7x96D3QBUms/m3wskoRwJlUc7WLAXx4Kaf44nZNNqjVtVQBigC8+fGra2DR/+Ns6ZY7S4JN9DZGcnV79iG6Ummi5gnOY4OhFc8HbNfRnv/Nf9e7UGj8LkzbcOMcrVxMjRBOXMp7jBIgiUoa89Oxh3Cmhhze0tIdfaNHOX3o2WAlVcT7/44sIfUm0aCcmeMX4JK0X
rsync.net ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKFjGEbhu9Spooi7N2N7IjoS67gaA0EFAV8xpEaxvNX13uorFjECq3dCL6UxwV/JXdr8A4LNcBrC4v0z88DSwp8=
rsync.net ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKNnX71CaVTNrMNGhrsqEhjqTvFp1UtXO1eUK61XX+p9

So it's possible that either:

  • Rclone's ssh implementation is either trying to use a key that's not in known_hosts; it's trying to use the ecdsa-sha2-nistp256 when it's only ssh-ed25519 that's available in the known_hosts file.
  • Rclone's ssh implementation can't do the ssh-ed25519 handshake, but can do the ecdsa-sha2-nistp256 handshake.

I'll try checking their other servers to see if there's one that doesn't use ssh-ed25519 . Maybe rclone can complete handshakes with those.

My recommendation: list all of them in the known_hosts file. If the Go SSH library is half-way sane it'll try all of them (that's what OpenSSH does, but I have a vague memory that the Go library isn't quite so smart). If it fails then just try them one at a time.

The key that's used will depend on the cipher suite negotiated.

correct about the base64 but your text did not work,
known_hosts:1: ssh: short read

i went to https://www.base64encode.org
jJCISHUXooV9/rh7kSeqiSuPvOVWBFCOe2CNkwTO46o
becomes
akpDSVNIVVhvb1Y5L3JoN2tTZXFpU3VQdk9WV0JGQ09lMkNOa3dUTzQ2bw==
and that works but get same error as OP.
so you solved my issue.

thanks much stranger :wink:

My recommendation: list all of them in the known_hosts file.

The problem is that for the server I was assigned (and for most of them in their fingerprint registry), there seems to be exactly just one key, according to ssh-keyscan.

Luckily, in the fingerprint registry I found another SFTP server with triple keys (sd1.rsync.net) and tried connecting to it. It turns out I get the same error only when the ecdsa-sha2-nistp256 is absent from the known_hosts file. So it does look like, at least in the multi-key case, Rclone is stuck using the ecdsa-sha2-nistp256.

I have no theory though on what happens when there's one and only one key. Maybe rclone is seeing more keys than ssh is? Can you check ssh hk-s020.rsync.net or ssh-keyscan hk-s020.rsync.net and check if your version of ssh is picking up other keys?

does this help?

19:09:24.195 Started a new SSH connection.
19:09:24.235 Connecting to SSH server usw-s003.rsync.net:22.
19:09:24.305 Connection established.
19:09:24.375 Server version: SSH-2.0-OpenSSH_8.2-hpn14v15 FreeBSD-openssh-portable-8.2.p1_1,1
19:09:24.375 First key exchange started. Cryptographic provider: Windows CNG (x86) with additions
19:09:24.535 Received host key from the server. Algorithm: Ed25519, size: 255 bits, SHA-256 fingerprint: jJCISHUXooV9/rh7kSeqiSuPvOVWBFCOe2CNkwTO46o.
19:09:24.555 First key exchange completed using Curve25519@libssh. Connection encryption and integrity: aes256-gcm, compression: none.
19:09:24.615 Attempting publickey authentication. Testing client key 'Profile 2' for acceptance.
19:09:24.726 The client key 'Profile 2' has been accepted.
19:09:24.726 Attempting publickey authentication. Signing with client key 'Profile 2'.
19:09:24.806 Authentication completed.

So I did a tcpdump of the traffic, and it looks like rclone and usw-s003 are negotiating a ssh-dss protocol. WTF... anyway...

I then ran ssh-keyscan -t ssh-dss usw-s003.... and got this result:

usw-s003.rsync.net ssh-dss AAAAB3NzaC1kc3MAAACBAM6bPzk4bTJZoF3OlcrAEJ7S7flFpLAoRfgIkRN2q0/xBVp/NV2r61f3s2FWUVasPv6FywOMKFJtzdEzgarB0UsLB7WbCyzk/7xt2mbRxEzzXP2K26CkZGpbFYRvuSev+XctVXNprpAprBshHDh0Vc2yjaE8Vp1MYShpvN64C6trAAAAFQCVBVdcSKN4+SRONMYsk9PlEIEvcwAAAIEAyz//dkcxu2bU8zDnjmGfdxorsOde/bd8kR35gTs31XEG7l0AudaDz+WzO/FDuRycBsrAPIWysuz5PfxDAwabtHAvrUUXj1c4G7mLTyoHQXeUnaNZP48u8LFyrqk/8iAW9H9+O69OfRxHJfW8Jmbx5MED5SNyNSbD0z0vwb9LdKEAAACAWNskanXdvtM9ScVKWc2wkr964+lGtW0qUXXfhEAS1bpire2UJ69A8Pp9pFQaldjiY0WbxkGr2PAlC+urS6o0RvbLfxkwnsjUaU5DOfgq+VW1ToZBh7futeYM9w83ZmRLzMFWb/0lMEHrvbuO7OeMHbTL0LkXyrcCBrgTQthFFjk=

And that seems to work!

For the kh-s020 server:

hk-s020.rsync.net ssh-dss AAAAB3NzaC1kc3MAAACBALmMGPP7QcC5CSyaGpo4J80eMPl0woucz21f0vCYW3FFRkKxrg7Lf8rOO462LKh9Wt78M0acldHeAacnm4T+qxynwNonQWJVHub/zW10eEnBCRspvCLM+WszoiK6DQGswrPY97N94wFnDpGd94uBO97JEQuM7KWpJvVMnwo7P2jFAAAAFQDIV2BClWyD49BSVDdnaiZdvahZCQAAAIAWq9joU5dl+KpFwLj5i6/+44cwCL/8sMu9aTQ73ezl2VyXo9vMKZFfZnRu2IIWD5HaKbbvxwRc8mIq3ydok6t8Iman3LtpwMOmYFFE0iIBEMRZ2ANyyPVQ1JeEHJqHcv9YSWeKLzLc7ahy0y9J0dt9JojG9JX5tXhC8Js3N7A5SwAAAIBAeBFssbFhrYf1r8TUmffSaSpxlqCA2aos7m/NENB1BbfxSpGVMemwU4CYIsSWKvYnilJD7ukiNz2lkrLb088vUz7c1Fwu5ND6Zan/0cGRvl/A4xLra32Ej7F8g8BMgHHRyC1ZMorGltgw3u+X311zXe6lLQoQJLjCCJHhY/gHjA==
1 Like

yes, that seems to work!

For the hk-s020 server:

Yep, that works! So there actually were hidden keys that ssh-keyscan wasn't picking up! Nice job on actually inspecting the traffic.

From the manpage

     -t type
             Specifies the type of the key to fetch from the scanned hosts.
             The possible values are ``rsa1'' for protocol version 1 and
             ``dsa'', ``ecdsa'', ``ed25519'', or ``rsa'' for protocol version
             2.  Multiple values may be specified by separating them with com-
             mas.  The default is to fetch ``rsa'', ``ecdsa'', and ``ed25519''
             keys.

thanks, i will take the credit, because that is the kind of person i am, but it was @sweh :upside_down_face:

Okay, so I guess we can use the dsa key for now. Although reading up on the topic, it looks like ssh-dsa is ideally supposed to be deprecated, with ed25519 as the new hotness.

Yeah; that's why ssh-keyscan didn't pick it up. Now why the Go ssh library decided that was the best cipher to pick, I dunno.

Maybe I'll create a PR to allow the cipher to be specified in the config file (since I think the library allows it). I'll stick it on my backlog of things to think about.

perhaps @rsync has a comment or suggestion?

Hmm.. https://github.com/golang/crypto/blob/master/ssh/common.go

var supportedHostKeyAlgos = []string{
	CertSigAlgoRSASHA2512v01, CertSigAlgoRSASHA2256v01,
	CertSigAlgoRSAv01, CertAlgoDSAv01, CertAlgoECDSA256v01,
	CertAlgoECDSA384v01, CertAlgoECDSA521v01, CertAlgoED25519v01,

	KeyAlgoECDSA256, KeyAlgoECDSA384, KeyAlgoECDSA521,
	SigAlgoRSASHA2512, SigAlgoRSASHA2256,
	SigAlgoRSA, KeyAlgoDSA,

	KeyAlgoED25519,
}

If I'm reading it correctly then this is saying that dsa (ssh-dss) is preferred over ed25519, which would explain what we're seeing. Huh.

They probably missed the "in preference order" comment and just appended KeyAlgoED25519 and CertAlgoED25519v01 to their groups.

If that is true, then it sounds like a bug, but I can't say I understand exactly what that code is doing!

You could try commenting out the ssh-dss entry in the go source and see what happens to see if your conjecture is correct.

Is there something we should add to the docs for --sftp-known-hosts-file here?