Question about handshake error

[Matt Caswell]

On 11/03/2020 15:31, Viktor Dukhovni wrote:
> On Wed, Mar 11, 2020 at 03:12:26PM +0000, Matt Caswell wrote:
> 
>>> The signature algorithm security level is not expected to be enforced
>>> on self-signed certificates (root CAs).  How is it happening here?
>>
>> It isn't. In this case the client is openssl but the server is unknown.
>> The problem is on the server side. The server is refusing to continue a
>> handshake where the sigalgs do not include sha1 because the server is
>> misconfigured to include a root in the cert chain which has a SHA1
>> signature. The server is obviously inspecting the mis-configured chain,
>> seeing the SHA1 signature, and giving up. This is not an OpenSSL problem.
>>
> 
> I think the server could be OpenSSL, because why I made sure that
> self-signed CA signatures are not subjected to security levels in
> x509_vfy.c, the same exclusion does not appear to be present in:
> 
>     int ssl_security_cert(SSL *s, SSL_CTX *ctx, X509 *x, int vfy, int is_ee)
>     {
>         if (vfy)
>             vfy = SSL_SECOP_PEER;
>         if (is_ee) {
>             if (!ssl_security_cert_key(s, ctx, x, SSL_SECOP_EE_KEY | vfy))
>                 return SSL_R_EE_KEY_TOO_SMALL;
>         } else {
>             if (!ssl_security_cert_key(s, ctx, x, SSL_SECOP_CA_KEY | vfy))
>                 return SSL_R_CA_KEY_TOO_SMALL;
>         }
>         if (!ssl_security_cert_sig(s, ctx, x, SSL_SECOP_CA_MD | vfy))
>             return SSL_R_CA_MD_TOO_WEAK;
>         return 1;
>     }
> 

The exclusion comes in ssl_security_cert_sig - so I think OpenSSL
behaves correctly:

static int ssl_security_cert_sig(SSL *s, SSL_CTX *ctx, X509 *x, int op)
{
    /* Lookup signature algorithm digest */
    int secbits, nid, pknid;
    /* Don't check signature if self signed */
    if ((X509_get_extension_flags(x) & EXFLAG_SS) != 0)
        return 1;
    if (!X509_get_signature_info(x, &nid, &pknid, &secbits, NULL))
        secbits = -1;
    /* If digest NID not defined use signature NID */
    if (nid == NID_undef)
        nid = pknid;
    if (s)
        return ssl_security(s, op, secbits, nid, x);
    else
        return ssl_ctx_security(ctx, op, secbits, nid, x);
}

Matt

> which servers use to check *their own* chains.  This function needs to
> exclude self-signed certificates from the final "cert_sig" check.
> 
> The server's own chain is not always anchored to a root CA, so just
> excluding the "top" cert (as when peer certs are verified in the
> X.509 code) is not quite the right thing to do here, instead we
> should check is self-signed flag, something along the lines of:
> 
>     if ((X509_get_extension_flags(x) & EXFLAG_SS) == 0
>         && !ssl_security_cert_sig(s, ctx, x, SSL_SECOP_CA_MD | vfy))
>             return SSL_R_CA_MD_TOO_WEAK;
>