[openssl-users] Can RSA_private_decrypt succeed with the wrong padding?

[Perrow, Graeme]

Apologies for the top-post; Outlook makes it hard to do anything else.

Here is a small C++ reproducible. I am generating a key pair, encrypting a small string using OAEP and decrypting using PKCS1 and expecting the decryption to fail.

If I run this (on 64-bit Red Hat 6) repeatedly, the program will eventually fail because RSA_private_decrypt doesn't fail. I can run it hundreds of times successfully before it fails. I have also seen it fail on Windows 7.

Graeme

#include <string.h>
#include <openssl/rsa.h>

int main( int, char ** )
/**********************/
{
    BIGNUM *exponent = BN_new();
    RSA *rsa_key = RSA_new();
    BN_set_word( exponent, RSA_F4 );
    int rc = RSA_generate_key_ex( rsa_key, 1024, exponent, NULL );
    if( rc == 0 ) {
	printf( "RSA key generation failed\n" );
	return 1;
    }
    
    int input_pad = RSA_PKCS1_OAEP_PADDING;
    const char *input = "abcd";
    size_t input_len = strlen( input );
    unsigned char encrypted[1000];
    unsigned char decrypted[1000];

    size_t enc_len = RSA_public_encrypt( (int)input_len, 
					 (const unsigned char *)input,
					 encrypted, rsa_key, input_pad );

    int output_pad = RSA_PKCS1_PADDING;
    memset( decrypted, 0, sizeof(decrypted) );
    size_t dec_len = RSA_private_decrypt( (int)enc_len, encrypted, decrypted,
					  rsa_key, output_pad );
    if( dec_len == -1 ) {
	return 0; // expected outcome
    }
    printf( "RSA_private_decrypt succeeded, len=%ld bytes\n", dec_len );
    return 1;
}

-----Original Message-----
From: openssl-users [mailto:openssl-users-bounces at openssl.org] On Behalf Of Viktor Dukhovni
Sent: Saturday, April 25, 2015 1:41 AM
To: openssl-users at openssl.org
Subject: Re: [openssl-users] Can RSA_private_decrypt succeed with the wrong padding?

On Sat, Apr 25, 2015 at 12:49:21AM +0000, Perrow, Graeme wrote:

> Using OpenSSL 1.0.1m on 64-bit Windows and Linux.
> 
> I have implemented RSA encryption using the RSA_public_encrypt and
> RSA_private_decrypt functions and various padding types. This is working
> fine except that in very rare cases, my test fails because decrypting
> succeeds when it should fail. I'm basically doing this (pseudocode):
> 
> RSA_public_encrypt( "abc", encrypted_data, my_public_key, RSA_PKCS1_OAEP_PADDING );
> RSA_private_decrypt( encrypted_data, decrypted_data, my_private_key, RSA_NO_PADDING );

A real code fragment would be substantially more useful that
"pseudo-code" here.  This should *always* succeed, provided you
pass the correct length to RSA_private_decrypt.  From the manpage:

       flen must be ... and exactly RSA_size(rsa) for RSA_NO_PADDING.

> Note that the padding types are different. The vast majority of the time,
> I get an error from the RSA_private_decrypt call but now and again, it
> succeeds.

You're doing something wrong, it should always recover the OAEP
padded data, which is basically random, you need to reverse the
OAEP padding to recover the plaintext.

> I don't understand the underlying crypto well enough to know - is it
> possible for RSA_private_decrypt to be unable to tell that the wrong
> padding was used, or is this a bug in OpenSSL?

When not using padding, that's not "wrong" padding, you're just
doing a raw RSA decrypt, and any input that is smaller than the
modulus (but has the same bit length) should decrypt to something.

-- 
	Viktor.
_______________________________________________
openssl-users mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users