ssl: Bad Certificate errors on perfectly valid SSL hosts

[1player]

Describe the bug

Many HTTPS hosts cannot be connected to because of error:

{:error,
 {:tls_alert,
  {:bad_certificate,
   ~c"TLS client: In state wait_cert_cr at ssl_handshake.erl:2113 generated CLIENT ALERT: Fatal - Bad Certificate\n"}}}

To Reproduce

From an Elixir shell:

:ssl.connect('api.smtp2go.com', 443, cacerts: :public_key.cacerts_get())

Also try with:

• www.kb.cert.org:443
• people.bath.ac.uk:443

Affected versions
At least 26.1.2 to 26.2.5 included.

Additional context

See also elixir-mint/mint#421

The maintainer of the Mint project explained this is caused by incorrect ordering of the certificate chain, but whatever the reason, other non-Erlang HTTPS libraries are perfectly happy with validating these websites, while Erlang fails because it's apparently too strict. Testing of these hosts with tools like https://www.ssllabs.com/ssltest/ presents no such validation issue.

I am building a web crawler that I've had to turn SSL verification off because of this very problem; the issue now is that hosts that once used to work, such as my transactional mail provider SMTP2GO at "api.smtp2go.com", after a certificate update might break and cause great disruption in production services. As it stands, I have been unable to use that service for the past month as Erlang decided the certificate is not valid anymore.

[1player]

Can reproduce with OTP 27 (elixir:1.17-otp-27)

CANNOT reproduce on OTP 25 (using image elixir:1.17-otp-25) because :ssl.connect does not set {:verify, :verify_peer} by default. When I add the option, fails with Fatal - Handshake Failure

[IngelaAndin]

When you verify the certificates you need to also supply appropriate other options for the verification. Like sha algorithm is not supported by default since OTP-26 . First example also needs to customize the host name check.

Erlang/OTP 27 [erts-15.0] [source-c183ff4f3f] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [jit:ns]

Eshell V15.0 (press Ctrl+G to abort, type help(). for help)
1> ssl:start().
ok
2> ssl:connect("api.smtp2go.com", 443, [{customize_hostname_check, [{match_fun, public_key:pkix_verify_hostname_match_fun(https)}]}, {signature_algs, ssl:signature_algs(all, 'tlsv1.3')}, {cacerts, public_key:cacerts_get()}]).
{ok,{sslsocket,{gen_tcp,#Port<0.5>,tls_connection,undefined},
[<0.123.0>,<0.122.0>]}}
3> ssl:connect("www.kb.cert.org", 443, [{signature_algs, ssl:signature_algs(all, 'tlsv1.3')}, {cacerts, public_key:cacerts_get()}]).
{ok,{sslsocket,{gen_tcp,#Port<0.6>,tls_connection,undefined},
[<0.128.0>,<0.127.0>]}}
4> ssl:connect("people.bath.ac.uk", 443, [{signature_algs, ssl:signature_algs(all, 'tlsv1.3')}, {cacerts, public_key:cacerts_get()}]).
{ok,{sslsocket,{gen_tcp,#Port<0.7>,tls_connection,undefined},
[<0.133.0>,<0.132.0>]}}
5>

Erlang/OTP 26 [erts-14.2.5] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [jit:ns]

Eshell V14.2.5 (press Ctrl+G to abort, type help(). for help)
1> ssl:start().
ok
2> ssl:connect("api.smtp2go.com", 443, [{customize_hostname_check, [{match_fun, public_key:pkix_verify_hostname_match_fun(https)}]}, {signature_algs, ssl:signature_algs(all, 'tlsv1.3')}, {cacerts, public_key:cacerts_get()}]).
{ok,{sslsocket,{gen_tcp,#Port<0.5>,tls_connection,undefined},
[<0.124.0>,<0.123.0>]}}
3> ssl:connect("www.kb.cert.org", 443, [{signature_algs, ssl:signature_algs(all, 'tlsv1.3')}, {cacerts, public_key:cacerts_get()}]).
{ok,{sslsocket,{gen_tcp,#Port<0.6>,tls_connection,undefined},
[<0.129.0>,<0.128.0>]}}
4> ssl:connect("people.bath.ac.uk", 443, [{signature_algs, ssl:signature_algs(all, 'tlsv1.3')}, {cacerts, public_key:cacerts_get()}]).
{ok,{sslsocket,{gen_tcp,#Port<0.7>,tls_connection,undefined},
[<0.134.0>,<0.133.0>]}}
5>

Note that you may not want to enable all non default signature algorithms. See User guide for guidance.

[1player]

Isn't it very premature to deprecate a signature algo (SHA1) that is still actively used and supported by 99% of the other clients, especially when there is no way to re-enable it globally? One might depend on third-party libraries that do not offer an easy way to tweak the signature algorithms because it is too low-level a concern, all you get is people enabling everything and making their software more insecure because of too strict a rule. Additionally, COMODO, arguably one of the biggest CA in the world still uses SHA1 signed certificates that expire in 2028. If OTP chooses to go against the grain compared to the rest of the world, I would love a config option to re-enable SHA globally. If there is such an option, the User Guide does not mention it, apart from patching and recompiling OTP, which is a massive pain.

…

On Mon, 17 Jun 2024, at 11:17, Ingela Andin wrote: When you verify the certificates you need to also supply appropriate other options for the verification. Like sha algorithm is not supported by default since OTP-26 . First example also needs to customize the host name check. Erlang/OTP 27 [erts-15.0] [source-c183ff4f3f] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [jit:ns] Eshell V15.0 (press Ctrl+G to abort, type help(). for help) 1> ssl:start(). ok 2> ssl:connect("api.smtp2go.com", 443, [{customize_hostname_check, [{match_fun, public_key:pkix_verify_hostname_match_fun(https)}]}, {signature_algs, ssl:signature_algs(all, 'tlsv1.3')}, {cacerts, public_key:cacerts_get()}]). {ok,{sslsocket,{gen_tcp,#Port<0.5>,tls_connection,undefined}, [<0.123.0>,<0.122.0>]}} 3> ssl:connect("www.kb.cert.org", 443, [{signature_algs, ssl:signature_algs(all, 'tlsv1.3')}, {cacerts, public_key:cacerts_get()}]). {ok,{sslsocket,{gen_tcp,#Port<0.6>,tls_connection,undefined}, [<0.128.0>,<0.127.0>]}} 4> ssl:connect("people.bath.ac.uk", 443, [{signature_algs, ssl:signature_algs(all, 'tlsv1.3')}, {cacerts, public_key:cacerts_get()}]). {ok,{sslsocket,{gen_tcp,#Port<0.7>,tls_connection,undefined}, [<0.133.0>,<0.132.0>]}} 5> Erlang/OTP 26 [erts-14.2.5] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [jit:ns] Eshell V14.2.5 (press Ctrl+G to abort, type help(). for help) 1> ssl:start(). ok 2> ssl:connect("api.smtp2go.com", 443, [{customize_hostname_check, [{match_fun, public_key:pkix_verify_hostname_match_fun(https)}]}, {signature_algs, ssl:signature_algs(all, 'tlsv1.3')}, {cacerts, public_key:cacerts_get()}]). {ok,{sslsocket,{gen_tcp,#Port<0.5>,tls_connection,undefined}, [<0.124.0>,<0.123.0>]}} 3> ssl:connect("www.kb.cert.org", 443, [{signature_algs, ssl:signature_algs(all, 'tlsv1.3')}, {cacerts, public_key:cacerts_get()}]). {ok,{sslsocket,{gen_tcp,#Port<0.6>,tls_connection,undefined}, [<0.129.0>,<0.128.0>]}} 4> ssl:connect("people.bath.ac.uk", 443, [{signature_algs, ssl:signature_algs(all, 'tlsv1.3')}, {cacerts, public_key:cacerts_get()}]). {ok,{sslsocket,{gen_tcp,#Port<0.7>,tls_connection,undefined}, [<0.134.0>,<0.133.0>]}} 5> Note that you may not want to enable all non default signature algorithms. See User guide for guidance. — Reply to this email directly, view it on GitHub <#8588 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAFIPSFUKGJJD5DAKP26U3LZH2ZSBAVCNFSM6AAAAABJNPBHJGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNZSHE3TKOBXHE>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[IngelaAndin]

I do not agree that it is premature not support it by default. It is not deprecated we just want the
user to make the security tradeoff. Come to think of it this is actually a good use for signature_algs_cert option that lets you support different algorithms for cert signing than for other uses of digital signatures in the protocol. And it would be much less serious to choose all algorithms here. But I think we could consider having a default for this one including {sha,rsa} for interoperability reasons it currently has no default value which in practice make it fallbacks to singnature_algs option.

[liamwhite]

Regarding the need to explicitly add {sha, rsa}, it seems like we cannot express the following constraint as ssl opts:

• Allow SHA1WithRSA for trusted root certificates in the chain
• Disallow SHA1WithRSA by default otherwise

This is what browsers and openssl s_client appear to do. Is there a possibility ssl can support this?

[IngelaAndin]

I will make this one open as I we are considering to make some interop change. I will investigate @liamwhite suggestion when back at work next week.

[IngelaAndin]

@liamwhite There is no need to explicitly configure that behavior that you are describing. It will work as

From TLS-1.3 RFC:

The signatures on certificates that are self-signed or certificates
   that are trust anchors are not validated, since they begin a
   certification path (see [[RFC5280], Section 3.2](https://datatracker.ietf.org/doc/html/rfc5280#section-3.2)).  A certificate that
   begins a certification path MAY use a signature algorithm that is not
   advertised as being supported in the "signature_algorithms"
   extension.

[liamwhite]

Okay. Is that describing the behavior that will be implemented in OTP 28, or how it is supposed to work in OTP 27?

[IngelaAndin]

@liamwhite this already works. The thing that we are considering changing is if there is an intermediate CA that is signed using sha1 we could allow that although not allowing sha1 for TLS protocol signatures, and that that should be default.

[1player]

I do believe that allowing intermediate SHA1 is probably the sanest approach here, given that they're still out there for a few years on a large number of Internet hosts.