[openssl-dev] Bug in pkey_rsa_encrypt() and _decrypt()

[Richard Levitte]

I think there's some confusion here...

OpenSSL's pkeyutl does indeed call something with out==NULL, but it's
not calling RSA_private_decrypt() or RSA_public_encrypt() directly,
it's calling the EVP_PKEY functions.  In *those* functions, there is a
check to see if the output argument is NULL and to return the
appropriate size in that case.  There is no promise that the lower
level functions called by the EVP_PKEY interface does the same.

As a matter of fact, the manual page for RSA_private_decrypt and
RSA_public_encrypt says this:

       ...
       RSA_public_encrypt() encrypts the flen bytes at from (usually a session
       key) using the public key rsa and stores the ciphertext in to. to must
       point to RSA_size(rsa) bytes of memory.
       ...
       RSA_private_decrypt() decrypts the flen bytes at from using the private
       key rsa and stores the plaintext in to. to must point to a memory
       section large enough to hold the decrypted data (which is smaller than
       RSA_size(rsa)). padding is the padding mode that was used to encrypt
       the data.
       ...

That should make it clear that a NULL output buffer isn't acceptable
at that level.

Cheers,
Richard

In message <A8BF4FBB-3E80-4126-9826-E9A64C46F9D0 at ll.mit.edu> on Tue, 26 Sep 2017 20:29:11 +0000, "Blumenthal, Uri - 0553 - MITLL" <uri at ll.mit.edu> said:

uri> Working on pkcs11 engine, I discovered a bug in crypto/rsa/rsa_pmeth.c in pkey_rsa_encrypt() and
uri> pkey_rsa_decrypt().
uri> 
uri> They cause a crash when called with out==NULL. Normally it should not happen – but when an
uri> engine is called, and it cannot process the padding – it reverts to the original OpenSSL-provided
uri> pkey_rsa_encrypt() or pkey_rsa_decrypt() (as appropriate). OpenSSL pkeyutl makes two calls when
uri> the key is not directly available (aka not presented in a disk file), and the first call with out==NULL
uri> crashes when RSA_private_decrypt() or RSA_public_encrypt() tries to copy the result to out.
uri> 
uri> The fix should be adding something like
uri> 
uri> if (out == NULL) {
uri> 
uri> int klen = RSA_size(ctx->pkey->pkey.rsa);
uri> 
uri> *outlen = klen;
uri> 
uri> return 1;
uri> 
uri> }
uri> 
uri> right before the call to RSA_public_encrypt().
uri> 
uri> P.S. It’s more critical in pkey_rsa_encrypt(), because it’s more likely that the engine would handle
uri> the decryption operation completely by itself.
uri> 
uri> --
uri> 
uri> Regards,
uri> 
uri> Uri Blumenthal
uri>