[openssl-users] openssl 1.1 certificate verification fails with non-standard public key algorithm

[Ken Goldman]

On 7/25/2018 10:47 AM, Viktor Dukhovni wrote:
> 
> 
>> On Jul 25, 2018, at 10:05 AM, Ken Goldman <kgoldman at us.ibm.com> wrote:
>>
>> I have a certificate with a non-standard public key algorithm -rsaesOaep.  See snippet #2.
>>
>> With openssl 1.0, I can validate  the certificate chain.  With openssl 1.1 it fails with the error X509_V_ERR_EE_KEY_TOO_SMALL.  See dump #1.
>>
>> I believe that this is due to new 1.1 code x509_vfy.c:check_key_level() calling X509_get0_pubkey().  That call will fail for the non-standard algorithm.
>>
>> The certificate is for old vendor hardware that cannot be updated.  What are my choices?
>>
>> - Remain on 1.0
>> - Some configuration option?
>> - Something else?
> 
> The immediate cause is the order of the checks in check_key_level().
> It first checks for a supported key, and only then short-circuits
> the logic at level <= 0 (my fault).  Perhaps level 0 should not be
> strict in this way, in which case we might reverse the order of
> then (pkey == NULL) and (level <= 0) tests:
> 
> static int check_key_level(X509_STORE_CTX *ctx, X509 *cert)
> {
>      EVP_PKEY *pkey = X509_get0_pubkey(cert);
>      int level = ctx->param->auth_level;
> 
>      /* Unsupported or malformed keys are not secure */
>      if (pkey == NULL)
>          return 0;
> 
>      if (level <= 0)
>          return 1;
>      if (level > NUM_AUTH_LEVELS)
>          level = NUM_AUTH_LEVELS;
> 
>      return EVP_PKEY_security_bits(pkey) >= minbits_table[level - 1];
> }

If you're suggesting that altering the above code to do the level check 
before the call to get pkey, I think that would fix my problem.

... if I can set level to a negative value.  How do I set level?  Is 
there an API or a configuration file.