[openssl-dev] rsautl.c incorrectly processes "-oaep" flag

[Douglas E Engert]

On 4/13/2017 4:18 PM, Richard Levitte wrote:
> In message <1EF605EC-D2DD-4D15-A27F-1E1CE7956BA9 at ll.mit.edu> on Thu, 13 Apr 2017 20:55:36 +0000, "Blumenthal, Uri - 0553 - MITLL" <uri at ll.mit.edu> said:
>
> uri> I am trying to use “openssl rsautl” to wrap/unwrap symmetric keys in a script. Decryption (and encryption too, but that isn’t relevant) is done using a token accessible via pkcs11 engine (libp11).
> uri>
> uri> The problem is: “rsautl” appears to assume that if “-oaep” flag is given, then the engine is going to handle OAEP padding. This is the screen log:
> uri>
> uri> $ openssl rsautl -engine pkcs11 -keyform ENGINE -encrypt -pubin -inkey "pkcs11:manufacturer=piv_II;object=KEY%20MAN%20pubkey;type=public" -oaep -in t256.dat -out t256.dat.enc
> uri> engine "pkcs11" set.
> uri> $ ls -l t256.dat.enc
> uri> -rw-r--r--  1 mouse   256 Apr 10 17:34 t256.dat.enc
> uri> $ openssl rsautl -engine pkcs11 -keyform ENGINE -decrypt -inkey "pkcs11:manufacturer=piv_II;object=KEY%20MAN%20key;type=private" -oaep -in t256.dat.enc -out t256.dat.dec
> uri> engine "pkcs11" set.
> uri> PKCS#11 token PIN:
> uri> PKCS#11: Unsupported padding type
> uri> RSA operation error
> uri> $
> uri>
> uri> libp11 does not know how to deal with OAEP padding, so it returns an error.
> uri>
> uri> Desired solution: in case of “-oaep” pass “RSA_NO_PADDING” to the engine (aka to libp11), and strip the padding using OpenSSL mechanisms.
> uri>
> uri> I’d like to see that fixed in both 1.1 and 1.0.2 branches.
>
> Wouldn't it be muuuuuch easier to add the following lines:
>
> 	case RSA_PKCS1_OAEP_PADDING:
> 		mechanism->mechanism = CKM_RSA_PKCS_OAEP;
> 		break;
>
> right about here?
> https://github.com/OpenSC/libp11/blob/master/src/p11_rsa.c#L72
>
> What you propose for OpenSSL is quite a lot harder to implement well,
> and one might also wonder why the OAEP padding should have that
> special treatment and no other?
>

Because there are parameters to the OAEP, and rsautl.c does not set it.

when not using an engine, rsa/rsa_pmeth.c in pkey_rsa_decrypt does something similar:

300 if (rctx->pad_mode == RSA_PKCS1_OAEP_PADDING) {

304         ret = RSA_private_decrypt(inlen, in, rctx->tbuf,
305                                   ctx->pkey->pkey.rsa, RSA_NO_PADDING);

312         ret = RSA_padding_check_PKCS1_OAEP_mgf1(out, ret, rctx->tbuf + i,
313                                                 ret - i, ret,
314                                                 rctx->oaep_label,
315                                                 rctx->oaep_labellen,
316                                                 rctx->md, rctx->mgf1md);




> Cheers,
> Richard
>

-- 

  Douglas E. Engert  <DEEngert at gmail.com>