Question about handshake error

[Viktor Dukhovni]

On Wed, Mar 11, 2020 at 03:12:26PM +0000, Matt Caswell wrote:

> > The signature algorithm security level is not expected to be enforced
> > on self-signed certificates (root CAs).  How is it happening here?
> 
> It isn't. In this case the client is openssl but the server is unknown.
> The problem is on the server side. The server is refusing to continue a
> handshake where the sigalgs do not include sha1 because the server is
> misconfigured to include a root in the cert chain which has a SHA1
> signature. The server is obviously inspecting the mis-configured chain,
> seeing the SHA1 signature, and giving up. This is not an OpenSSL problem.
> 

I think the server could be OpenSSL, because why I made sure that
self-signed CA signatures are not subjected to security levels in
x509_vfy.c, the same exclusion does not appear to be present in:

    int ssl_security_cert(SSL *s, SSL_CTX *ctx, X509 *x, int vfy, int is_ee)
    {
        if (vfy)
            vfy = SSL_SECOP_PEER;
        if (is_ee) {
            if (!ssl_security_cert_key(s, ctx, x, SSL_SECOP_EE_KEY | vfy))
                return SSL_R_EE_KEY_TOO_SMALL;
        } else {
            if (!ssl_security_cert_key(s, ctx, x, SSL_SECOP_CA_KEY | vfy))
                return SSL_R_CA_KEY_TOO_SMALL;
        }
        if (!ssl_security_cert_sig(s, ctx, x, SSL_SECOP_CA_MD | vfy))
            return SSL_R_CA_MD_TOO_WEAK;
        return 1;
    }

which servers use to check *their own* chains.  This function needs to
exclude self-signed certificates from the final "cert_sig" check.

The server's own chain is not always anchored to a root CA, so just
excluding the "top" cert (as when peer certs are verified in the
X.509 code) is not quite the right thing to do here, instead we
should check is self-signed flag, something along the lines of:

    if ((X509_get_extension_flags(x) & EXFLAG_SS) == 0
        && !ssl_security_cert_sig(s, ctx, x, SSL_SECOP_CA_MD | vfy))
            return SSL_R_CA_MD_TOO_WEAK;

-- 
    Viktor.