[openssl-users] sendmail, openssl 1.1.1, tls1.3

[Viktor Dukhovni]

> On Oct 15, 2018, at 7:49 PM, Carl Byington <carl at five-ten-sg.com> wrote:
> 
>> Perhaps Sendmail is setting the CA names the client side, and then
>> OpenSSL is trying to serialize the names of all your CAs to the
>> server.  This is a bad idea.  Don't do that.  Try using CApath, and
>> no or an explicitly empty CAfile, and see if that helps.
> 
> Do you mean CACertFile and CACertPath? Redhat/Centos stock
> sendmail.mc/cf uses:
> 
> O CACertFile=/etc/pki/tls/certs/ca-bundle.crt
> O CACertPath=/etc/pki/tls/certs

Yes, these.  It is common practice with TLS servers to use the CAfile
(in Sendmail aka CACertFile) as the source of CA names to send to clients
in certificate requests (when those are enabled).  Most MTAs don't request
client certificates, so this "overloading" of the parameter is mostly
harmless.  Still, in Postfix the documentation recommends keeping the
CAfile short (or even not specified at all), and having any CA cerficates
used for validating client certs be specified via CApath.  CApath is used
for indexed lookup and not enumeration.

With TLS 1.3, you suddenly have clients optionally soliciting certificates
by specific CA from servers, and if you use the same SSL_CTX for both the
client and server roles, the same CA list will be sent to remote servers
that you'd use to solicit certificates from clients.

My advice is to make sure that the CAfile is mercifully short or not
set at all (assuming that's possible in Sendmail without disabling TLS).

> pointing the CACertFile to 750KB file with 149 certificates.

With 149 certs, and typical CA names O(80) bytes, we're looking at
~12KB of cert names, which should fit into an extension that can be
up to 64KB in size.  So overflowing the extension size limit would
not be my first guess.  If you make the CA bundle available (send
it to me off-list?) I can take a closer look.

> I have a
> private CA that signed my sendmail certificate, so using:
> 
> O CACertFile=/etc/pki/tls/certs/my-ca-certificate.pem
> O CACertPath=/etc/pki/tls/certs
> O ServerSSLOptions=+SSL_OP_NO_SSLv2 +SSL_OP_NO_SSLv3
> +SSL_OP_CIPHER_SERVER_PREFERENCE
> O ClientSSLOptions=+SSL_OP_NO_SSLv2 +SSL_OP_NO_SSLv3
> 
> on both the client and server sendmail machines, we get TLSv1.3 !

So the there's something toxic about the CA list as a whole.
Either indeed some problem certiificate, or a problem encoding
the lot of them?

>> Do you see any calls to SSL_CTX_set0_CA_list()?
> 
> No, but there is a call to  SSL_CTX_set_client_CA_list(*ctx,
> SSL_load_client_CA_file(cacertfile)) which would read that ca-bundle.crt
> file.

Right, but I think that the underlying list is the same for client->server
as for server->client.  So this has the unfortunate effect of making
the client fill its HELLO message with an enormous list of CA names.

A separate SSL_CTX for the client would make Sendmail less prone to
such mishaps.  Try Postfix some time, we don't have that issue... :-)

-- 
	Viktor.