[openssl-dev] Bug in pkey_rsa_encrypt() and _decrypt()

[Dr. Stephen Henson]

On Tue, Sep 26, 2017, Blumenthal, Uri - 0553 - MITLL wrote:

> Working on pkcs11 engine, I discovered a bug in crypto/rsa/rsa_pmeth.c in pkey_rsa_encrypt() and pkey_rsa_decrypt().
> 
> They cause a crash when called with out==NULL. Normally it should not happen ??? but when an engine is called, and it cannot process the padding ??? it reverts to the original OpenSSL-provided pkey_rsa_encrypt() or pkey_rsa_decrypt() (as appropriate). OpenSSL pkeyutl makes two calls when the key is not directly available (aka not presented in a disk file), and the first call with out==NULL crashes when RSA_private_decrypt() or RSA_public_encrypt() tries to copy the result to out.
> 

The original RSA pkey method has the flag EVP_PKEY_FLAG_AUTOARGLEN set which
handles the NULL output automatically so it is not handled in pkey_rsa_*().

The ENGINE should either set this flag itself too or deal with NULL arguments
manually if that is not appropriate.

Steve.
--
Dr Stephen N. Henson. OpenSSL project core developer.
Commercial tech support now available see: http://www.openssl.org