[openssl-dev] FIPS / RSA / ENGINE bug?

[Hong Cho]

Hi,

I generated OpenSSL libcrypto (1.0.1l) with the OpenSSL FIPS crypto module
(2.0.8) on FreeBSD 8.4 amd64.

It seems to build fine, and with OPENSSL_FIPS, it seems to behave correctly
(e.g., MD5 is refused, DH with 512-bit key is refused, etc.).

However, genrsa(1) is failing with the following message.

------
# /usr/bin/openssl genrsa -out rsa1 2048
Generating RSA private key, 2048 bit long modulus
34374116264:error:0409B09D:rsa routines:RSA_generate_key_ex:non fips rsa
method:[...]/openssl/crypto/rsa/rsa_gen.c:88:
------

So, I put some debugging printf's, and this is what I found.

------
# /usr/bin/openssl genrsa -out rsa1 2048
XXX MAIN: engine = 0x0
Generating RSA private key, 2048 bit long modulus
XXX MAIN: e = 0x0
XXX RSA_get_default_method: returning FIPS_rsa_pkcs1_ssleay()
XXX RSA_new_method: ret->meth = 0x800b4a7e0
XXX RSA_new_method: engine = 0x0
XXX RSA_new_method: ret->engine = 0x800e28100
XXX RSA_new_method: ret->meth = 0x800b2ee40
XXX MAIN: rsa->meth = 0x800b2ee40
XXX RSA_generate_key_ex: rsa->meth = 0x800b2ee40
34374116264:error:0409B09D:rsa routines:RSA_generate_key_ex:non fips rsa
method:/usr/home/hongch/ns_depot/TOT/usr.src/crypto/openssl/crypto/rsa/rsa_gen.c:88:
------

​So, it seems like the FIPS RSA_METHOD gets overridden by the default
ENGINE (the machine does not have any hardware crypto module), which does
not have​ the RSA_FLAG_FIPS_METHOD flag set.

I understand that I can rebuild the library with "./config no-engine", but
I haven't seen this mentioned in the FIPS guide.

Is this the right way to do it? Or can the interaction between FIPS and
(the default) ENGINE be better?

Thanks.

Hong.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mta.openssl.org/pipermail/openssl-dev/attachments/20150227/7b4739d8/attachment.html>