[openssl-users] openssl 1.1 certificate verification fails with non-standard public key algorithm

[Ken Goldman]

Seeking advice.

I have a certificate with a non-standard public key algorithm 
-rsaesOaep.  See snippet #2.

With openssl 1.0, I can validate  the certificate chain.  With openssl 
1.1 it fails with the error X509_V_ERR_EE_KEY_TOO_SMALL.  See dump #1.

I believe that this is due to new 1.1 code x509_vfy.c:check_key_level() 
calling X509_get0_pubkey().  That call will fail for the non-standard 
algorithm.

The certificate is for old vendor hardware that cannot be updated.  What 
are my choices?

- Remain on 1.0
- Some configuration option?
- Something else?


#1 ~~~~~~~~~

openssl verify -CAfile cafile.pem infcert.pem

error 66 at 0 depth lookup: EE certificate key too weak
error infcert.pem: verification failed
22794983405376:error:0609E09C:digital envelope 
routines:pkey_set_type:unsupported algorithm:crypto/evp/p_lib.c:206:
22794983405376:error:0B09406F:x509 certificate 
routines:x509_pubkey_decode:unsupported 
algorithm:crypto/x509/x_pubkey.c:113:

#2 ~~~~~~~~~

         Subject:
         Subject Public Key Info:
             Public Key Algorithm: rsaesOaep
             Unable to load Public Key
140619228055400:error:0609E09C:digital envelope 
routines:PKEY_SET_TYPE:unsupported algorithm:p_lib.c:239:
140619228055400:error:0B07706F:x509 certificate 
routines:X509_PUBKEY_get:unsupported algorithm:x_pubkey.c:155:
         X509v3 extensions: