[openssl-users] Can RSA_private_decrypt succeed with the wrong padding?

[Viktor Dukhovni]

On Sat, Apr 25, 2015 at 12:49:21AM +0000, Perrow, Graeme wrote:

> Using OpenSSL 1.0.1m on 64-bit Windows and Linux.
> 
> I have implemented RSA encryption using the RSA_public_encrypt and
> RSA_private_decrypt functions and various padding types. This is working
> fine except that in very rare cases, my test fails because decrypting
> succeeds when it should fail. I'm basically doing this (pseudocode):
> 
> RSA_public_encrypt( "abc", encrypted_data, my_public_key, RSA_PKCS1_OAEP_PADDING );
> RSA_private_decrypt( encrypted_data, decrypted_data, my_private_key, RSA_NO_PADDING );

A real code fragment would be substantially more useful that
"pseudo-code" here.  This should *always* succeed, provided you
pass the correct length to RSA_private_decrypt.  From the manpage:

       flen must be ... and exactly RSA_size(rsa) for RSA_NO_PADDING.

> Note that the padding types are different. The vast majority of the time,
> I get an error from the RSA_private_decrypt call but now and again, it
> succeeds.

You're doing something wrong, it should always recover the OAEP
padded data, which is basically random, you need to reverse the
OAEP padding to recover the plaintext.

> I don't understand the underlying crypto well enough to know - is it
> possible for RSA_private_decrypt to be unable to tell that the wrong
> padding was used, or is this a bug in OpenSSL?

When not using padding, that's not "wrong" padding, you're just
doing a raw RSA decrypt, and any input that is smaller than the
modulus (but has the same bit length) should decrypt to something.

-- 
	Viktor.