[openssl-dev] rsautl.c incorrectly processes "-oaep" flag

[Richard Levitte]

In message <1EF605EC-D2DD-4D15-A27F-1E1CE7956BA9 at ll.mit.edu> on Thu, 13 Apr 2017 20:55:36 +0000, "Blumenthal, Uri - 0553 - MITLL" <uri at ll.mit.edu> said:

uri> I am trying to use “openssl rsautl” to wrap/unwrap symmetric keys in a script. Decryption (and encryption too, but that isn’t relevant) is done using a token accessible via pkcs11 engine (libp11).
uri> 
uri> The problem is: “rsautl” appears to assume that if “-oaep” flag is given, then the engine is going to handle OAEP padding. This is the screen log:
uri> 
uri> $ openssl rsautl -engine pkcs11 -keyform ENGINE -encrypt -pubin -inkey "pkcs11:manufacturer=piv_II;object=KEY%20MAN%20pubkey;type=public" -oaep -in t256.dat -out t256.dat.enc
uri> engine "pkcs11" set.
uri> $ ls -l t256.dat.enc 
uri> -rw-r--r--  1 mouse   256 Apr 10 17:34 t256.dat.enc
uri> $ openssl rsautl -engine pkcs11 -keyform ENGINE -decrypt -inkey "pkcs11:manufacturer=piv_II;object=KEY%20MAN%20key;type=private" -oaep -in t256.dat.enc -out t256.dat.dec
uri> engine "pkcs11" set.
uri> PKCS#11 token PIN: 
uri> PKCS#11: Unsupported padding type
uri> RSA operation error
uri> $
uri> 
uri> libp11 does not know how to deal with OAEP padding, so it returns an error.
uri> 
uri> Desired solution: in case of “-oaep” pass “RSA_NO_PADDING” to the engine (aka to libp11), and strip the padding using OpenSSL mechanisms.
uri> 
uri> I’d like to see that fixed in both 1.1 and 1.0.2 branches.

Wouldn't it be muuuuuch easier to add the following lines:

	case RSA_PKCS1_OAEP_PADDING:
		mechanism->mechanism = CKM_RSA_PKCS_OAEP;
		break;

right about here?  
https://github.com/OpenSC/libp11/blob/master/src/p11_rsa.c#L72

What you propose for OpenSSL is quite a lot harder to implement well,
and one might also wonder why the OAEP padding should have that
special treatment and no other?

Cheers,
Richard

-- 
Richard Levitte         levitte at openssl.org
OpenSSL Project         http://www.openssl.org/~levitte/