[openssl-users] sendmail, openssl 1.1.1, tls1.3

[Viktor Dukhovni]

On Mon, Oct 15, 2018 at 10:42:26AM -0700, Carl Byington wrote:

> I have a build of sendmail with openssl 1.1.1. It can deliver to
> localhost via tls1.3, but nowhere else.
> 
> STARTTLS=client, error: connect failed=-1, reason=internal error,
> SSL_error=1, errno=0, retry=-1
> 
> STARTTLS=client: error:14228044:SSL routines:construct_ca_names:internal error:ssl/statem/statem_lib.c:2289:

This fails when the list of acceptable CAs sent to the peer contains
a NULL name or the ASN.1 encoding routines fail, or memory is not
available to encode the name into the output packet.

    if (ca_sk != NULL) {
        int i;

        for (i = 0; i < sk_X509_NAME_num(ca_sk); i++) {
            unsigned char *namebytes;
            X509_NAME *name = sk_X509_NAME_value(ca_sk, i);
            int namelen;

            if (name == NULL
                    || (namelen = i2d_X509_NAME(name, NULL)) < 0
                    || !WPACKET_sub_allocate_bytes_u16(pkt, namelen,
                                                       &namebytes)
                    || i2d_X509_NAME(name, &namebytes) != namelen) {
                SSLfatal(s, SSL_AD_INTERNAL_ERROR, SSL_F_CONSTRUCT_CA_NAMES,
                         ERR_R_INTERNAL_ERROR);
                return 0;
            }
        }
    }

> It works correctly if I disable tls1.3 via:
> 
> O ServerSSLOptions=+SSL_OP_NO_SSLv2 +SSL_OP_NO_SSLv3 +SSL_OP_NO_TLSv1_3 +SSL_OP_CIPHER_SERVER_PREFERENCE
> O ClientSSLOptions=+SSL_OP_NO_SSLv2 +SSL_OP_NO_SSLv3 +SSL_OP_NO_TLSv1_3

Unlike TLS 1.2, in TLS 1.3 the client MAY send a list of acceptable
certificate authorities to the server as an extension in its client
HELLO: https://tools.ietf.org/html/rfc8446#section-4.2.4

Perhaps Sendmail is setting the CA names the client side, and then
OpenSSL is trying to serialize the names of all your CAs to the
server.  This is a bad idea.  Don't do that.  Try using CApath, and
no or an explicitly empty CAfile, and see if that helps.

> Is this another symptom of
> https://github.com/openssl/openssl/issues/7384, or is there something
> else going on here?

No something else.  A pointer to source code of the Sendmail in
question may be helpful.  Do you see any calls to SSL_CTX_set0_CA_list()?

I guess Sendmail being both a client and server in one, may be using
a single SSL context for both purposes, and may therefore end up
configuring the CA list for both.

If you use an empty CA file, Sendmail will probably not leak all
your trusted CA names with every connection, and be more reliable
as well.

I suspect some sort of Sendmail-specific misuse of the API, but
it is also possible that use of CA names on the client side is
not sufficiently well tested on the OpenSSL side.

-- 
	Viktor.