[openssl-users] Can RSA_private_decrypt succeed with the wrong padding?

[Perrow, Graeme]

Using OpenSSL 1.0.1m on 64-bit Windows and Linux.

I have implemented RSA encryption using the RSA_public_encrypt and RSA_private_decrypt functions and various padding types. This is working fine except that in very rare cases, my test fails because decrypting succeeds when it should fail. I'm basically doing this (pseudocode):

RSA_public_encrypt( "abc", encrypted_data, my_public_key, RSA_PKCS1_OAEP_PADDING );
RSA_private_decrypt( encrypted_data, decrypted_data, my_private_key, RSA_NO_PADDING );

Note that the padding types are different. The vast majority of the time, I get an error from the RSA_private_decrypt call but now and again, it succeeds. The output data is NOT "abc" however, it's seemingly random data. I've seen varying lengths for the output data as well, from 11 bytes to 115.

I don't understand the underlying crypto well enough to know - is it possible for RSA_private_decrypt to be unable to tell that the wrong padding was used, or is this a bug in OpenSSL?

Thanks
Graeme Perrow


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mta.openssl.org/pipermail/openssl-users/attachments/20150425/dca647de/attachment.html>