Remote error: tls: handshake failure with Dell ECS Storage

elysch · April 24, 2021, 1:05am

What is the problem you are having with rclone?

Not being able to ls to Dell ECS Storage (S3) - remote error: tls: handshake failure

What is your rclone version (output from `rclone version`)

rclone v1.55.0

os/type: linux
os/arch: amd64
go/version: go1.16.2
go/linking: static
go/tags: cmount

Which OS you are using and how many bits (eg Windows 7, 64 bit)

CentOS Linux release 7.7.1908 (Core)
Linux XXXXX 3.10.0-1062.9.1.el7.x86_64 #1 SMP Fri Dec 6 15:49:49 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Which cloud storage system are you using? (eg Google Drive)

Dell ECS

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

rclone -vv ls ecs:XXXXXXX

The rclone config contents with secrets removed.

[ecs]
type = s3
provider = Other
env_auth = false
access_key_id = XXXXXXXXXX
secret_access_key = XXXXXXXXXXXXX
endpoint = XXXXXXXX
acl = private

A log from the command with the `-vv` flag

[XXXXXX@XXXXXX ~]$ rclone -vv ls ecs:/YYYYYYYY/ZZZZZZZZZ
2021/04/23 19:38:28 DEBUG : Using config file from "/XXXXXXXX/.config/rclone/rclone.conf"
2021/04/23 19:38:28 DEBUG : rclone: Version "v1.55.0" starting with parameters ["rclone" "-vv" "ls" "ecs:/YYYYYYYY/ZZZZZZZZ"]
2021/04/23 19:38:28 DEBUG : Creating backend with remote "ecs:/YYYYYYY/ZZZZZZZ"
2021/04/23 19:38:28 DEBUG : fs cache: renaming cache item "ecs:/YYYYYYYY/ZZZZZZZ" to be canonical "ecs:YYYYYYYY/ZZZZZZZZ"
2021/04/23 19:38:28 DEBUG : 2 go routines active
2021/04/23 19:38:28 Failed to ls: RequestError: send request failed
caused by: Get "https://XXXXXXX/YYYYYYYYYY?delimiter=&max-keys=1000&prefix=ZZZZZZZZ%2F": remote error: tls: handshake failure

I tried with different buckets, getting the same result.

Found at htt ps://forum.rclone.org/t/tls-version-for-rclone-1-50-1/15018 rclone uses go to connect

I found at htt ps://stackoverflow.com/questions/41250665/go-https-client-issue-remote-error-tls-handshake-failure someone with the same error message so I tried with openssl, and got:

$ openssl s_client -connect XXXXXXXX:443
...
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : DHE-RSA-AES256-GCM-SHA384
    Session-ID: 608333D0B43B84A06B296217791ABE8DE5D81B4185954FB4F8D8421D193C8A4C
    Session-ID-ctx:
    Master-Key: XXXXXXXXX
    Key-Arg   : None
    Krb5 Principal: None
    PSK identity: None
    PSK identity hint: None
    Start Time: 1619211216
    Timeout   : 300 (sec)
    Verify return code: 18 (self signed certificate)
---

And the solution talks something about the cipher.

So, looked for the ciphers implemented in go. And found them at htt ps://golang.org/pkg/crypto/tls/ and couldn't find the one mentioned by openssl

Looked for the supported ciphers so found a script that gave me the ones supported by openssl:

[XXXXXX@XXXXXXX ~]$ ./test_ciphers XXXXXXXX:443
Obtaining cipher list from OpenSSL 1.0.2k-fips 26 Jan 2017.
Testing ECDHE-RSA-AES256-GCM-SHA384...NO (sslv3 alert handshake failure)
Testing ECDHE-ECDSA-AES256-GCM-SHA384...NO (sslv3 alert handshake failure)
Testing ECDHE-RSA-AES256-SHA384...NO (sslv3 alert handshake failure)
Testing ECDHE-ECDSA-AES256-SHA384...NO (sslv3 alert handshake failure)
Testing ECDHE-RSA-AES256-SHA...NO (sslv3 alert handshake failure)
Testing ECDHE-ECDSA-AES256-SHA...NO (sslv3 alert handshake failure)
Testing DH-DSS-AES256-GCM-SHA384...NO (sslv3 alert handshake failure)
Testing DHE-DSS-AES256-GCM-SHA384...NO (sslv3 alert handshake failure)
Testing DH-RSA-AES256-GCM-SHA384...NO (sslv3 alert handshake failure)
Testing DHE-RSA-AES256-GCM-SHA384...YES
Testing DHE-RSA-AES256-SHA256...NO (sslv3 alert handshake failure)
Testing DHE-DSS-AES256-SHA256...NO (sslv3 alert handshake failure)
Testing DH-RSA-AES256-SHA256...NO (sslv3 alert handshake failure)
Testing DH-DSS-AES256-SHA256...NO (sslv3 alert handshake failure)
Testing DHE-RSA-AES256-SHA...NO (sslv3 alert handshake failure)
Testing DHE-DSS-AES256-SHA...NO (sslv3 alert handshake failure)
Testing DH-RSA-AES256-SHA...NO (sslv3 alert handshake failure)
Testing DH-DSS-AES256-SHA...NO (sslv3 alert handshake failure)
Testing DHE-RSA-CAMELLIA256-SHA...NO (sslv3 alert handshake failure)
Testing DHE-DSS-CAMELLIA256-SHA...NO (sslv3 alert handshake failure)
Testing DH-RSA-CAMELLIA256-SHA...NO (sslv3 alert handshake failure)
Testing DH-DSS-CAMELLIA256-SHA...NO (sslv3 alert handshake failure)
Testing AECDH-AES256-SHA...NO (sslv3 alert handshake failure)
Testing ADH-AES256-GCM-SHA384...NO (sslv3 alert handshake failure)
Testing ADH-AES256-SHA256...NO (sslv3 alert handshake failure)
Testing ADH-AES256-SHA...NO (sslv3 alert handshake failure)
Testing ADH-CAMELLIA256-SHA...NO (sslv3 alert handshake failure)
Testing ECDH-RSA-AES256-GCM-SHA384...NO (sslv3 alert handshake failure)
Testing ECDH-ECDSA-AES256-GCM-SHA384...NO (sslv3 alert handshake failure)
Testing ECDH-RSA-AES256-SHA384...NO (sslv3 alert handshake failure)
Testing ECDH-ECDSA-AES256-SHA384...NO (sslv3 alert handshake failure)
Testing ECDH-RSA-AES256-SHA...NO (sslv3 alert handshake failure)
Testing ECDH-ECDSA-AES256-SHA...NO (sslv3 alert handshake failure)
Testing AES256-GCM-SHA384...NO (sslv3 alert handshake failure)
Testing AES256-SHA256...NO (sslv3 alert handshake failure)
Testing AES256-SHA...NO (sslv3 alert handshake failure)
Testing CAMELLIA256-SHA...NO (sslv3 alert handshake failure)
Testing PSK-AES256-CBC-SHA...NO (no ciphers available)
Testing ECDHE-RSA-AES128-GCM-SHA256...NO (sslv3 alert handshake failure)
Testing ECDHE-ECDSA-AES128-GCM-SHA256...NO (sslv3 alert handshake failure)
Testing ECDHE-RSA-AES128-SHA256...NO (sslv3 alert handshake failure)
Testing ECDHE-ECDSA-AES128-SHA256...NO (sslv3 alert handshake failure)
Testing ECDHE-RSA-AES128-SHA...NO (sslv3 alert handshake failure)
Testing ECDHE-ECDSA-AES128-SHA...NO (sslv3 alert handshake failure)
Testing DH-DSS-AES128-GCM-SHA256...NO (sslv3 alert handshake failure)
Testing DHE-DSS-AES128-GCM-SHA256...NO (sslv3 alert handshake failure)
Testing DH-RSA-AES128-GCM-SHA256...NO (sslv3 alert handshake failure)
Testing DHE-RSA-AES128-GCM-SHA256...YES
Testing DHE-RSA-AES128-SHA256...NO (sslv3 alert handshake failure)
Testing DHE-DSS-AES128-SHA256...NO (sslv3 alert handshake failure)
Testing DH-RSA-AES128-SHA256...NO (sslv3 alert handshake failure)
Testing DH-DSS-AES128-SHA256...NO (sslv3 alert handshake failure)
Testing DHE-RSA-AES128-SHA...NO (sslv3 alert handshake failure)
Testing DHE-DSS-AES128-SHA...NO (sslv3 alert handshake failure)
Testing DH-RSA-AES128-SHA...NO (sslv3 alert handshake failure)
Testing DH-DSS-AES128-SHA...NO (sslv3 alert handshake failure)
Testing DHE-RSA-SEED-SHA...NO (sslv3 alert handshake failure)
Testing DHE-DSS-SEED-SHA...NO (sslv3 alert handshake failure)
Testing DH-RSA-SEED-SHA...NO (sslv3 alert handshake failure)
Testing DH-DSS-SEED-SHA...NO (sslv3 alert handshake failure)
Testing DHE-RSA-CAMELLIA128-SHA...NO (sslv3 alert handshake failure)
Testing DHE-DSS-CAMELLIA128-SHA...NO (sslv3 alert handshake failure)
Testing DH-RSA-CAMELLIA128-SHA...NO (sslv3 alert handshake failure)
Testing DH-DSS-CAMELLIA128-SHA...NO (sslv3 alert handshake failure)
Testing AECDH-AES128-SHA...NO (sslv3 alert handshake failure)
Testing ADH-AES128-GCM-SHA256...NO (sslv3 alert handshake failure)
Testing ADH-AES128-SHA256...NO (sslv3 alert handshake failure)
Testing ADH-AES128-SHA...NO (sslv3 alert handshake failure)
Testing ADH-SEED-SHA...NO (sslv3 alert handshake failure)
Testing ADH-CAMELLIA128-SHA...NO (sslv3 alert handshake failure)
Testing ECDH-RSA-AES128-GCM-SHA256...NO (sslv3 alert handshake failure)
Testing ECDH-ECDSA-AES128-GCM-SHA256...NO (sslv3 alert handshake failure)
Testing ECDH-RSA-AES128-SHA256...NO (sslv3 alert handshake failure)
Testing ECDH-ECDSA-AES128-SHA256...NO (sslv3 alert handshake failure)
Testing ECDH-RSA-AES128-SHA...NO (sslv3 alert handshake failure)
Testing ECDH-ECDSA-AES128-SHA...NO (sslv3 alert handshake failure)
Testing AES128-GCM-SHA256...NO (sslv3 alert handshake failure)
Testing AES128-SHA256...NO (sslv3 alert handshake failure)
Testing AES128-SHA...NO (sslv3 alert handshake failure)
Testing SEED-SHA...NO (sslv3 alert handshake failure)
Testing CAMELLIA128-SHA...NO (sslv3 alert handshake failure)
Testing PSK-AES128-CBC-SHA...NO (no ciphers available)
Testing ECDHE-RSA-DES-CBC3-SHA...NO (sslv3 alert handshake failure)
Testing ECDHE-ECDSA-DES-CBC3-SHA...NO (sslv3 alert handshake failure)
Testing EDH-RSA-DES-CBC3-SHA...NO (sslv3 alert handshake failure)
Testing EDH-DSS-DES-CBC3-SHA...NO (sslv3 alert handshake failure)
Testing DH-RSA-DES-CBC3-SHA...NO (sslv3 alert handshake failure)
Testing DH-DSS-DES-CBC3-SHA...NO (sslv3 alert handshake failure)
Testing AECDH-DES-CBC3-SHA...NO (sslv3 alert handshake failure)
Testing ADH-DES-CBC3-SHA...NO (sslv3 alert handshake failure)
Testing ECDH-RSA-DES-CBC3-SHA...NO (sslv3 alert handshake failure)
Testing ECDH-ECDSA-DES-CBC3-SHA...NO (sslv3 alert handshake failure)
Testing DES-CBC3-SHA...NO (sslv3 alert handshake failure)
Testing IDEA-CBC-SHA...NO (sslv3 alert handshake failure)
Testing PSK-3DES-EDE-CBC-SHA...NO (no ciphers available)
Testing KRB5-IDEA-CBC-SHA...NO (no ciphers available)
Testing KRB5-DES-CBC3-SHA...NO (no ciphers available)
Testing KRB5-IDEA-CBC-MD5...NO (no ciphers available)
Testing KRB5-DES-CBC3-MD5...NO (no ciphers available)
Testing ECDHE-RSA-RC4-SHA...NO (sslv3 alert handshake failure)
Testing ECDHE-ECDSA-RC4-SHA...NO (sslv3 alert handshake failure)
Testing AECDH-RC4-SHA...NO (sslv3 alert handshake failure)
Testing ADH-RC4-MD5...NO (sslv3 alert handshake failure)
Testing ECDH-RSA-RC4-SHA...NO (sslv3 alert handshake failure)
Testing ECDH-ECDSA-RC4-SHA...NO (sslv3 alert handshake failure)
Testing RC4-SHA...NO (sslv3 alert handshake failure)
Testing RC4-MD5...NO (sslv3 alert handshake failure)
Testing PSK-RC4-SHA...NO (no ciphers available)
Testing KRB5-RC4-SHA...NO (no ciphers available)
Testing KRB5-RC4-MD5...NO (no ciphers available)
Testing ECDHE-RSA-NULL-SHA...NO (sslv3 alert handshake failure)
Testing ECDHE-ECDSA-NULL-SHA...NO (sslv3 alert handshake failure)
Testing AECDH-NULL-SHA...NO (sslv3 alert handshake failure)
Testing ECDH-RSA-NULL-SHA...NO (sslv3 alert handshake failure)
Testing ECDH-ECDSA-NULL-SHA...NO (sslv3 alert handshake failure)
Testing NULL-SHA256...NO (sslv3 alert handshake failure)
Testing NULL-SHA...NO (sslv3 alert handshake failure)
Testing NULL-MD5...NO (sslv3 alert handshake failure)

Found at htt ps://testssl.sh/openssl-iana.mapping.html a way to translate openssl cipher names to iana names, and didn't help since the ones reported by openssl, converted to iana, where also not found.

I think go doesn't include the ciphers I need.

Could someone help me find a solution to be able to connect to our object storage?

Thank you in advance.

ncw · April 24, 2021, 10:32pm

That cipher is in the go test suite

        test = &clientTest{
                name:   "ClientCert-RSA-AES256-GCM-SHA384",
                args:   []string{"-cipher", "ECDHE-RSA-AES256-GCM-SHA384", "-Verify", "1"},
                config: config,
                cert:   testRSACertificate,
                key:    testRSAPrivateKey,
        }

        runClientTestTLS12(t, test)

So I'm guessing it is supported.

The self signed certificate will be a problem though, try adding

  --no-check-certificate   Do not verify the server SSL certificate. Insecure.

elysch · April 26, 2021, 3:56pm

Still no luck with that.

[XXXXX@XXXXX ~]$ rclone --no-check-certificate -vv lsd ecs:
2021/04/26 10:37:20 DEBUG : Using config file from "/XXXXXXX/.config/rclone/rclone.conf"
2021/04/26 10:37:20 DEBUG : rclone: Version "v1.55.0" starting with parameters ["rclone" "--no-check-certificate" "-vv" "lsd" "ecs:"]
2021/04/26 10:37:20 DEBUG : Creating backend with remote "ecs:"
2021/04/26 10:37:20 ERROR : : error listing: RequestError: send request failed
caused by: Get "htt ps://XXXXXXX/": remote error: tls: handshake failure
2021/04/26 10:37:20 DEBUG : 2 go routines active
2021/04/26 10:37:20 Failed to lsd with 2 errors: last error was: RequestError: send request failed
caused by: Get "htt ps://XXXXXXXX/": remote error: tls: handshake failure

What else can I try?

The buckets in our ECS storage, have namespaces defined. Maybe there is a way to specify them?.

ncw · April 26, 2021, 9:00pm

Can you get a tcpdump of the exchange?

Can you try rclone copyurl on the URL in question? And then with --no-check-certificate

Not sure what this means.

sweh · April 26, 2021, 10:15pm

Careful; the original was DHE-RSA-AES256-GCM-SHA384 which is cipher suite 0x9f but you referenced ECDHE-RSA-AES256-GCM-SHA384 which is 0xc030.

I don't see 9f defined in crypto/tls/cipher_suites.go so I'm not sure that's supported by GoLang.

EDIT: GoLang devels say they have no desire to support DHE; crypto/tls: add DHE support · Issue #7758 · golang/go · GitHub

ncw · April 27, 2021, 1:54pm

Well spotted.

@elysch maybe you could run one of the ssl scan tools of script on the server to see exactly what SSL/TLS options it supports?

elysch · April 28, 2021, 6:24pm

I only found a script to test the storage against openssl supported cyphers and It showed only two:

elysch:

[XXXXXX@XXXXXXX ~]$ ./test_ciphers XXXXXXXX:443
Obtaining cipher list from OpenSSL 1.0.2k-fips 26 Jan 2017.
...
Testing DHE-RSA-AES256-GCM-SHA384...YES
...
Testing DHE-RSA-AES128-GCM-SHA256...YES
...

Do you happen to have another script to try?

ncw · April 28, 2021, 9:35pm

I found those two ciphers here

TLS_DHE_RSA_WITH_AES_256_GCM_SHA384
- 0x00, 0x9F
TLS_DHE_RSA_WITH_AES_128_GCM_SHA256
- 0x00, 0x9E

I double checked in the go source and these are definitely not supported, as suggested by @sweh in the comment above.

So I think unless you can persuade whoever administers the Dell ECS storage to install some more modern ciphers we are out of luck.

The Go team are quite aggressive at not supporting weak ciphers. I don't think DHE-RSA-AES256-GCM-SHA384 is considered weak, but it has flaws as you'll see if you read the page above.

Possible solutions would be to

use a fork of the go TLS library (I'm not keen on this)
Use an ssl proxy
get the admins to enable some more ciphers

sweh · April 28, 2021, 10:10pm

The main problem with DHE is the key size; a 1024bit DHE key is considered weak; you want at least 2048bits. This has compatibility issues and performance issues. F5 BigIP won't support 2048bit keys because of this. (I wrote a blog post about this last year; More modern TLS settings · Ramblings of a Unix Geek )

ncw · April 29, 2021, 6:53am

Great article thanks. That explains the go team's rationale nicely. From an rclone perspective I'd rather the ciphers were available and could be opted into, but that wasn't the path the go team went down.

elysch · April 30, 2021, 12:23am

Yes... great article. Thank you for sharing.

Finally Got it working. Thank you all for the time spent.

We modified the Dell ECS load balancer SSL configuration.

elysch · April 30, 2021, 1:06am

Some aditional comments:

found and installed the sslscan:

sudo yum install sslscan

Since we re using a self signed certificate, as sugested, we had to use
--no-check-certificate

I was just about doing the tcpdump capture, when I decided to dig down the balancer configuration. I'm not really sure why it wasn't working, but what fixed it was to enable SSL-acceleration and reencrypt. When selected, it extended the available properties and permitted to select a "Cipher Set" among other settings

What I tried to say about namespaces that are defined in our Dell ECS Storage is that the bucket configuration allows to define "groups" of buckets in a defined "namespace". Those buckets can use the same access credentials (or not). In my desperation I began to think the problem may had something to do with that.

I really appreciate all your help.

Ely

ncw · April 30, 2021, 8:03am

I'm glad you got it working, and thanks for the writeup for future readers.

I think that probably the Go team's decision on ciphers was a good one since this is the first problem like this I can remember on the forum and you got it working in the end

sweh · April 30, 2021, 11:53am

FWIW, I prefer nmap for my scans:

nmap -p 443 hostname --script +ssl-enum-ciphers

For my own host:

% nmap -p 443 www.sweharris.org --script +ssl-enum-ciphers

Starting Nmap 6.40 ( http://nmap.org ) at 2021-04-30 07:50 EDT
Nmap scan report for www.sweharris.org (66.228.55.57)
Host is up (0.040s latency).
rDNS record for 66.228.55.57: linode.spuddy.org
PORT    STATE SERVICE
443/tcp open  https
| ssl-enum-ciphers: 
|   SSLv3: No supported ciphers found
|   TLSv1.2: 
|     ciphers: 
|       TLS_DHE_RSA_WITH_AES_128_GCM_SHA256 - strong
|       TLS_DHE_RSA_WITH_AES_256_GCM_SHA384 - strong
|       TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 - strong
|       TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 - strong
|       TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 - strong
|       TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 - strong
|     compressors: 
|       NULL
|_  least strength: strong

Nmap done: 1 IP address (1 host up) scanned in 1.78 seconds

Which nicely tells me only TLS1.2 is supported (so no TLS1.1, TLS1.0) and I only have the 6 ciphers. Which matches my blog post, fortunately!

ncw · April 30, 2021, 5:57pm

I tried that but it didn't work

$ nmap -p 443 rclone.org --script +ssl-enum-ciphers
Starting Nmap 7.80 ( https://nmap.org ) at 2021-04-30 18:56 BST
Nmap scan report for rclone.org (5.153.250.7)
Host is up (0.0079s latency).
Other addresses for rclone.org (not scanned): 2a02:24e0:8:61f9::1
rDNS record for 5.153.250.7: nick3.miniserver.com

PORT    STATE SERVICE
443/tcp open  https

Nmap done: 1 IP address (1 host up) scanned in 0.63 seconds

Do I need to install the scripts?

elysch · April 30, 2021, 5:58pm

Yes. I agree.

Thanks again

sweh · April 30, 2021, 6:30pm

The scripts are there otherwise you'd have got an error telling you it's missing.

Interesting; I get the same response when doing it on rclone.org. I did an nmap -sV .... and the response included "Bad request - Client send a HTTP request to an HTTPS server" messages.

I did a network dump without the -sV flag and didn't get any error in the text. But nothing useful; no handhake completed, no cert presented. Odd!

Do you have a firewall or filtering device in front of the server that's blocking the request? That can break things; you do need a clear network path

ncw · April 30, 2021, 7:22pm

It is behind a very dumb firewall (linux iptables) - I don't think that should affect things. Certainly other servers behind the same firewall do work.

It is caddy doing the serving - I expect that is the difference.

system · June 30, 2021, 3:23pm

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.