[openssl-users] PKEY signing failing in fips mode

[Mahoda Ratnayaka]

Hi,

I'm trying to change the ssh-rsa.c to be fips compliant. So, after some
investigation I added the following code to to ssh_rsa_sign function to
make it fips compliant.

==========================================================================
signing_key = EVP_PKEY_new();
EVP_PKEY_assign_RSA(signing_key, key->rsa);

ctx = EVP_PKEY_CTX_new(signing_key, NULL /* no engine */);
EVP_PKEY_sign_init(ctx);
EVP_PKEY_CTX_set_rsa_padding(ctx, RSA_PKCS1_PADDING);
EVP_PKEY_CTX_set_signature_md(ctx, EVP_sha256());
EVP_PKEY_sign(ctx, sig, &slen, digest, sizeof(digest));
============================================================================

I also, tried changing  the code to be as follows:
=========================================================================
+
+ EVP_MD_CTX_init(&mctx);
+ EVP_SignInit_ex(&mctx, EVP_sha256 (), NULL);
+ EVP_SignUpdate(&mctx, data, datalen);

  slen = RSA_size(key->rsa);
  sig = xmalloc(slen);

- ok = RSA_sign(nid, digest, dlen, sig, &len, key->rsa);
+ EVP_SignFinal(&mctx, sig, &len, pkey);
===========================================================================

But, unfortunately both these approaches end with the following error
message.
"error:0408E09E:rsa routines:PKEY_RSA_SIGN:operation not allowed in fips
mode."

It would be much appreciated if anyone can let me know why I'm hitting
this, and if there is any way of getting around it.

Thanks,
Mahoda
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mta.openssl.org/pipermail/openssl-users/attachments/20151204/d39adf81/attachment.html>