[openssl-dev] Bug in pkey_rsa_encrypt() and _decrypt()

[Blumenthal, Uri - 0553 - MITLL]

Working on pkcs11 engine, I discovered a bug in crypto/rsa/rsa_pmeth.c in pkey_rsa_encrypt() and pkey_rsa_decrypt().

 

They cause a crash when called with out==NULL. Normally it should not happen – but when an engine is called, and it cannot process the padding – it reverts to the original OpenSSL-provided pkey_rsa_encrypt() or pkey_rsa_decrypt() (as appropriate). OpenSSL pkeyutl makes two calls when the key is not directly available (aka not presented in a disk file), and the first call with out==NULL crashes when RSA_private_decrypt() or RSA_public_encrypt() tries to copy the result to out.

 

The fix should be adding something like

 

  if (out == NULL) {

       int klen = RSA_size(ctx->pkey->pkey.rsa);

       *outlen = klen;

       return 1;

  }

 

right before the call to RSA_public_encrypt().

 

P.S. It’s more critical in pkey_rsa_encrypt(), because it’s more likely that the engine would handle the decryption operation completely by itself.

--

Regards,

Uri Blumenthal

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mta.openssl.org/pipermail/openssl-dev/attachments/20170926/f1fef4d8/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 5211 bytes
Desc: not available
URL: <http://mta.openssl.org/pipermail/openssl-dev/attachments/20170926/f1fef4d8/attachment.bin>