[openssl-dev] Bug in pkey_rsa_encrypt() and _decrypt()

[Dr. Stephen Henson]

On Wed, Sep 27, 2017, Blumenthal, Uri - 0553 - MITLL wrote:

>     > Working on pkcs11 engine, I discovered a bug in crypto/rsa/rsa_pmeth.c in pkey_rsa_encrypt() and pkey_rsa_decrypt().
>     > 
>     > They cause a crash when called with out==NULL. Normally it should not happen 
>     > but when an engine is called, and it cannot process the padding it reverts to the
>     > original OpenSSL-provided pkey_rsa_encrypt() or pkey_rsa_decrypt() (as appropriate).
> 
>     The original RSA pkey method has the flag EVP_PKEY_FLAG_AUTOARGLEN set which
>     handles the NULL output automatically so it is not handled in pkey_rsa_*().
>     
>     The ENGINE should either set this flag itself too or deal with NULL arguments
>     manually if that is not appropriate.
>     
> Since hardware tokens I???m dealing with do not perform any public key operations (the engine in this case is used to merely pull and provide the public key  to the requestor) I???m somewhat ambivalent about writing engine Encrypt function specifically for handling the NULL argument case. On the one hand, it???s the simplest solution, and it avoids going through OpenSSL modification process.;) On the other hand, it???s not as clean as I???d like.
> 
> Where would I set this flag ? And would it work when the public key is on the token, and needs to be retrieved via engine?

It's set when the method is created via EVP_PKEY_meth_new(). If you set it it
assumes the public key components are set in the EVP_PKEY and calls
EVP_PKEY_size() appropriately to handle the NULL argument and if the supplied
buffer is too small.

Checkout the M_check_autoarg macro in crypto/evp/pmeth_fn.c to see exactly
what it does.

Steve.
--
Dr Stephen N. Henson. OpenSSL project core developer.
Commercial tech support now available see: http://www.openssl.org