RSA_METHOD.rsa_sign not called in FIPS mode

[Paul O'Keefe]

I'm using an OpenSSL engine that uses the RSA_FLAG_SIGN_VER flag and implements RSA_METHOD.rsa_sign() instead rsa_priv_enc(). This is mainly because of the requirement that it work with Windows CryptoAPI which does not support low-level RSA signing (see CAPI engine). Everything works as it should until FIPS mode is enabled. Under FIPS mode, the "non-implemented" rsa_priv_enc() is called and an error is returned. The simplified backtrace is:

#0  rsa_priv_enc // non-implemented engine function
#1  FIPS_rsa_sign_digest // FIPS canister
#2  pkey_rsa_sign
#3  EVP_SignFinal

It appears that FIPS_rsa_sign_digest() never checks RSA_FLAG_SIGN_VER or calls rsa_sign() - it simply defaults to rsa_priv_enc(). I can't find any place rsa_sign is called.

There are posts that specifically reference running CAPI with FIPS mode, so I don't know what I'm missing.
http://openssl.6102.n7.nabble.com/FIPS-with-CAPI-Engine-td26273.html

Using OpenSSL 1.0.2o and FIPS canister 2.0.2 (older but I checked the latest release and it behaves the same).

Thank you.
Paul

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mta.openssl.org/pipermail/openssl-users/attachments/20201109/caf85630/attachment-0001.html>