[openssl-dev] rsautl.c incorrectly processes "-oaep" flag

[Blumenthal, Uri - 0553 - MITLL]

I am trying to use “openssl rsautl” to wrap/unwrap symmetric keys in a script. Decryption (and encryption too, but that isn’t relevant) is done using a token accessible via pkcs11 engine (libp11).

The problem is: “rsautl” appears to assume that if “-oaep” flag is given, then the engine is going to handle OAEP padding. This is the screen log:

$ openssl rsautl -engine pkcs11 -keyform ENGINE -encrypt -pubin -inkey "pkcs11:manufacturer=piv_II;object=KEY%20MAN%20pubkey;type=public" -oaep -in t256.dat -out t256.dat.enc
engine "pkcs11" set.
$ ls -l t256.dat.enc 
-rw-r--r--  1 mouse   256 Apr 10 17:34 t256.dat.enc
$ openssl rsautl -engine pkcs11 -keyform ENGINE -decrypt -inkey "pkcs11:manufacturer=piv_II;object=KEY%20MAN%20key;type=private" -oaep -in t256.dat.enc -out t256.dat.dec
engine "pkcs11" set.
PKCS#11 token PIN: 
PKCS#11: Unsupported padding type
RSA operation error
$

libp11 does not know how to deal with OAEP padding, so it returns an error.

Desired solution: in case of “-oaep” pass “RSA_NO_PADDING” to the engine (aka to libp11), and strip the padding using OpenSSL mechanisms.

I’d like to see that fixed in both 1.1 and 1.0.2 branches.
— 
Regards,
Uri

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 5211 bytes
Desc: not available
URL: <http://mta.openssl.org/pipermail/openssl-dev/attachments/20170413/bb1e2a57/attachment.bin>