[openssl-dev] rsautl.c incorrectly processes "-oaep" flag

[Richard Levitte]

In message <006b8116-8aad-18f6-8759-2696ebf38de4 at gmail.com> on Thu, 13 Apr 2017 16:41:35 -0500, Douglas E Engert <deengert at gmail.com> said:

deengert> 
deengert> 
deengert> On 4/13/2017 4:18 PM, Richard Levitte wrote:
deengert> > In message <1EF605EC-D2DD-4D15-A27F-1E1CE7956BA9 at ll.mit.edu> on Thu,
deengert> > 13 Apr 2017 20:55:36 +0000, "Blumenthal, Uri - 0553 - MITLL"
deengert> > <uri at ll.mit.edu> said:
deengert> >
deengert> > uri> I am trying to use “openssl rsautl” to wrap/unwrap symmetric keys
deengert> > in a script. Decryption (and encryption too, but that isn’t relevant)
deengert> > is done using a token accessible via pkcs11 engine (libp11).
deengert> > uri>
deengert> > uri> The problem is: “rsautl” appears to assume that if “-oaep” flag
deengert> > is given, then the engine is going to handle OAEP padding. This is the
deengert> > screen log:
deengert> > uri>
deengert> > uri> $ openssl rsautl -engine pkcs11 -keyform ENGINE -encrypt -pubin
deengert> > -inkey
deengert> > "pkcs11:manufacturer=piv_II;object=KEY%20MAN%20pubkey;type=public"
deengert> > -oaep -in t256.dat -out t256.dat.enc
deengert> > uri> engine "pkcs11" set.
deengert> > uri> $ ls -l t256.dat.enc
deengert> > uri> -rw-r--r--  1 mouse   256 Apr 10 17:34 t256.dat.enc
deengert> > uri> $ openssl rsautl -engine pkcs11 -keyform ENGINE -decrypt -inkey
deengert> > "pkcs11:manufacturer=piv_II;object=KEY%20MAN%20key;type=private" -oaep
deengert> > -in t256.dat.enc -out t256.dat.dec
deengert> > uri> engine "pkcs11" set.
deengert> > uri> PKCS#11 token PIN:
deengert> > uri> PKCS#11: Unsupported padding type
deengert> > uri> RSA operation error
deengert> > uri> $
deengert> > uri>
deengert> > uri> libp11 does not know how to deal with OAEP padding, so it returns
deengert> > an error.
deengert> > uri>
deengert> > uri> Desired solution: in case of “-oaep” pass “RSA_NO_PADDING” to the
deengert> > engine (aka to libp11), and strip the padding using OpenSSL
deengert> > mechanisms.
deengert> > uri>
deengert> > uri> I’d like to see that fixed in both 1.1 and 1.0.2 branches.
deengert> >
deengert> > Wouldn't it be muuuuuch easier to add the following lines:
deengert> >
deengert> > 	case RSA_PKCS1_OAEP_PADDING:
deengert> > 		mechanism->mechanism = CKM_RSA_PKCS_OAEP;
deengert> > 		break;
deengert> >
deengert> > right about here?
deengert> > https://github.com/OpenSC/libp11/blob/master/src/p11_rsa.c#L72
deengert> >
deengert> > What you propose for OpenSSL is quite a lot harder to implement well,
deengert> > and one might also wonder why the OAEP padding should have that
deengert> > special treatment and no other?
deengert> >
deengert> 
deengert> Because there are parameters to the OAEP, and rsautl.c does not set
deengert> it.
deengert> 
deengert> when not using an engine, rsa/rsa_pmeth.c in pkey_rsa_decrypt does
deengert> something similar:
deengert> 
deengert> 300 if (rctx->pad_mode == RSA_PKCS1_OAEP_PADDING) {
deengert> 
deengert> 304         ret = RSA_private_decrypt(inlen, in, rctx->tbuf,
deengert> 305 ctx->pkey->pkey.rsa, RSA_NO_PADDING);
deengert> 
deengert> 312 ret = RSA_padding_check_PKCS1_OAEP_mgf1(out, ret, rctx->tbuf + i,
deengert> 313                                                 ret - i, ret,
deengert> 314                                                 rctx->oaep_label,
deengert> 315 rctx->oaep_labellen,
deengert> 316 rctx->md, rctx->mgf1md);

Good point.  But then, rsautl is a poor choice, as it uses the RSA
API.  For something more general and with a whole lot more
functionality, pkeyutl is the better choice.

Incidently, for decryption, it will end up calling exactly the code
you're citing, and with -pkeyopt, you can specify the padding mode and
its necessary data.

Cheers,
Richard

-- 
Richard Levitte         levitte at openssl.org
OpenSSL Project         http://www.openssl.org/~levitte/