[openssl-dev] Bug in pkey_rsa_encrypt() and _decrypt()

[Blumenthal, Uri - 0553 - MITLL]

    > Working on pkcs11 engine, I discovered a bug in crypto/rsa/rsa_pmeth.c in pkey_rsa_encrypt() and pkey_rsa_decrypt().
    > 
    > They cause a crash when called with out==NULL. Normally it should not happen 
    > but when an engine is called, and it cannot process the padding it reverts to the
    > original OpenSSL-provided pkey_rsa_encrypt() or pkey_rsa_decrypt() (as appropriate).

    The original RSA pkey method has the flag EVP_PKEY_FLAG_AUTOARGLEN set which
    handles the NULL output automatically so it is not handled in pkey_rsa_*().
    
    The ENGINE should either set this flag itself too or deal with NULL arguments
    manually if that is not appropriate.
    
Since hardware tokens I’m dealing with do not perform any public key operations (the engine in this case is used to merely pull and provide the public key  to the requestor) I’m somewhat ambivalent about writing engine Encrypt function specifically for handling the NULL argument case. On the one hand, it’s the simplest solution, and it avoids going through OpenSSL modification process.;) On the other hand, it’s not as clean as I’d like.

Where would I set this flag ? And would it work when the public key is on the token, and needs to be retrieved via engine?
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 5211 bytes
Desc: not available
URL: <http://mta.openssl.org/pipermail/openssl-dev/attachments/20170927/5e0011da/attachment.bin>