[openssl-users] openssl 1.1 certificate verification fails with non-standard public key algorithm

[Viktor Dukhovni]

> On Jul 25, 2018, at 10:05 AM, Ken Goldman <kgoldman at us.ibm.com> wrote:
> 
> I have a certificate with a non-standard public key algorithm -rsaesOaep.  See snippet #2.
> 
> With openssl 1.0, I can validate  the certificate chain.  With openssl 1.1 it fails with the error X509_V_ERR_EE_KEY_TOO_SMALL.  See dump #1.
> 
> I believe that this is due to new 1.1 code x509_vfy.c:check_key_level() calling X509_get0_pubkey().  That call will fail for the non-standard algorithm.
> 
> The certificate is for old vendor hardware that cannot be updated.  What are my choices?
> 
> - Remain on 1.0
> - Some configuration option?
> - Something else?

The immediate cause is the order of the checks in check_key_level().
It first checks for a supported key, and only then short-circuits
the logic at level <= 0 (my fault).  Perhaps level 0 should not be
strict in this way, in which case we might reverse the order of
then (pkey == NULL) and (level <= 0) tests:

static int check_key_level(X509_STORE_CTX *ctx, X509 *cert)
{
    EVP_PKEY *pkey = X509_get0_pubkey(cert);
    int level = ctx->param->auth_level;

    /* Unsupported or malformed keys are not secure */
    if (pkey == NULL)
        return 0;

    if (level <= 0)
        return 1;
    if (level > NUM_AUTH_LEVELS)
        level = NUM_AUTH_LEVELS;

    return EVP_PKEY_security_bits(pkey) >= minbits_table[level - 1];
}

-- 
	Viktor.