[openssl-dev] rsautl.c incorrectly processes "-oaep" flag

[Blumenthal, Uri - 0553 - MITLL]

On 4/13/17, 5:18 PM, "Richard Levitte" <levitte at openssl.org> wrote:

    uri> . . . . .
    uri> libp11 does not know how to deal with OAEP padding, so it returns an error.
    uri> 
    uri> Desired solution: in case of “-oaep” pass “RSA_NO_PADDING” to the engine (aka to libp11), and strip the padding using OpenSSL mechanisms.
    uri> 
    uri> I’d like to see that fixed in both 1.1 and 1.0.2 branches.
    
    Wouldn't it be muuuuuch easier to add the following lines [to libp11/src/p11_rsa.c]:
    
    	case RSA_PKCS1_OAEP_PADDING:
    		mechanism->mechanism = CKM_RSA_PKCS_OAEP;
    		break;
    

I’m afraid not – because currently OpenSSL does have full support for OAEP, and OpenSC has none. This is what causes the problem: OpenSSL expects the engine (libp11 and OpenSC) to handle OAEP, which they cannot do.

    What you propose for OpenSSL is quite a lot harder to implement well,

I agree that it’s harder to implement *well*, but it is a lot simpler and shorter to implement in rsautl.c (a few lines of code), as compared to adding the whole support for OAEP to OpenSC (which – I agree – would be great to have, but let’s be realistic: it’s not there now).

    and one might also wonder why the OAEP padding should have that
    special treatment and no other?

I’d say the same treatment is applicable to any padding that is supported by OpenSSL but not by (the majority of) PKCS#11 devices (and OpenSC). 

What OpenSSL does programmatically with this is (IMHO) perfect. This code works correctly with the token that only does raw RSA (the original had a lot more of error checking stuff ():

	privkey = ENGINE_load_private_key(e, KeyManPrivKey, NULL, &cb_data);

	ctx = EVP_PKEY_CTX_new(privkey, NULL);
	EVP_PKEY_free(privkey);

	rv = EVP_PKEY_decrypt_init(ctx);
	if (rv <= 0) goto end;
	rv = EVP_PKEY_CTX_set_rsa_padding(ctx, PADDING);

	*olen = 0;
	rv = EVP_PKEY_decrypt(ctx, NULL, olen, in, inlen);

	*out = OPENSSL_malloc(*olen);
	rv = EVP_PKEY_decrypt(ctx, *out, olen, in, inlen);
    end:

Perhaps rsautl.c could do the same? Instead of what it’s doing now (aka calling RSA_private_decrypt())?
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 5211 bytes
Desc: not available
URL: <http://mta.openssl.org/pipermail/openssl-dev/attachments/20170413/ba33ac8c/attachment-0001.bin>