[openssl-dev] FIPS / RSA / ENGINE bug?

[Dr. Stephen Henson]

On Fri, Feb 27, 2015, Hong Cho wrote:

> Hi,
> 
> I generated OpenSSL libcrypto (1.0.1l) with the OpenSSL FIPS crypto module
> (2.0.8) on FreeBSD 8.4 amd64.
> 
> It seems to build fine, and with OPENSSL_FIPS, it seems to behave correctly
> (e.g., MD5 is refused, DH with 512-bit key is refused, etc.).
> 
> However, genrsa(1) is failing with the following message.
> 
> ------
> # /usr/bin/openssl genrsa -out rsa1 2048
> Generating RSA private key, 2048 bit long modulus
> 34374116264:error:0409B09D:rsa routines:RSA_generate_key_ex:non fips rsa
> method:[...]/openssl/crypto/rsa/rsa_gen.c:88:
> ------
> 
> So, I put some debugging printf's, and this is what I found.
> 
> ------
> # /usr/bin/openssl genrsa -out rsa1 2048
> XXX MAIN: engine = 0x0
> Generating RSA private key, 2048 bit long modulus
> XXX MAIN: e = 0x0
> XXX RSA_get_default_method: returning FIPS_rsa_pkcs1_ssleay()
> XXX RSA_new_method: ret->meth = 0x800b4a7e0
> XXX RSA_new_method: engine = 0x0
> XXX RSA_new_method: ret->engine = 0x800e28100
> XXX RSA_new_method: ret->meth = 0x800b2ee40
> XXX MAIN: rsa->meth = 0x800b2ee40
> XXX RSA_generate_key_ex: rsa->meth = 0x800b2ee40
> 34374116264:error:0409B09D:rsa routines:RSA_generate_key_ex:non fips rsa
> method:/usr/home/hongch/ns_depot/TOT/usr.src/crypto/openssl/crypto/rsa/rsa_gen.c:88:
> ------
> 
> ???So, it seems like the FIPS RSA_METHOD gets overridden by the default
> ENGINE (the machine does not have any hardware crypto module), which does
> not have??? the RSA_FLAG_FIPS_METHOD flag set.
> 
> I understand that I can rebuild the library with "./config no-engine", but
> I haven't seen this mentioned in the FIPS guide.
> 
> Is this the right way to do it? Or can the interaction between FIPS and
> (the default) ENGINE be better?
> 

Can you work out which ENGINE it is that is doing that? If you print out
ENGINE_get_id(engine) that will show it. I'm guessing it's the cryptodev
ENGINE?

Using no-engine is the only way I can immediately think of to address this
without modifying OpenSSL or making some API calls to disable the ENGINE.

Steve.
--
Dr Stephen N. Henson. OpenSSL project core developer.
Commercial tech support now available see: http://www.openssl.org