[openssl-dev] Bug: digest parameter is rejected

[Blumenthal, Uri - 0553 - MITLL]

OpenSSL implementation of OAEP wrongly refuses to set the hash algorithm, preventing one from using SHA-2 family:


You'll probably need to pick up master and its -rsa_mgf1_md argument to pkeyutl.


Thank you – better with “-pkeyopt rsa_mgf1_md:sha256”. But still broken – as it affects only the MGF1 setting, but not the hash setting. I’d say it still needs to allow “-pkeyutl digest:xxx” parameter.

 

$ ~/openssl-1.1/bin/openssl version

OpenSSL 1.1.1-dev  xx XXX xxxx

$ ~/openssl-1.1/bin/openssl pkeyutl -encrypt -in t1264.dat -out t1264.dat.enc2.oaep -keyform DER -pubin -inkey rsa3072pub.der -pkeyopt rsa_padding_mode:oaep -pkeyopt rsa_mgf1_md:sha256

$ yhsm2-tool --decrypt -m RSA-PKCS-OAEP --id 0301 -i t1264.dat.enc2.oaep -o t1264.dat.dec2 --hash-algorithm SHA256

Using slot 0 with a present token (0x0)

Logging in to "YubiHSM".

Please enter User PIN: 

Using decrypt algorithm RSA-PKCS-OAEP

OAEP parameters: hashAlg=SHA256, mgf=MGF1-SHA256, source_type=0, source_ptr=0x0, source_len=0

error: PKCS11 function C_Decrypt failed: rv = CKR_FUNCTION_FAILED (0x6)

Aborting.

$ yhsm2-tool --decrypt -m RSA-PKCS-OAEP --id 0301 -i t1264.dat.enc2.oaep -o t1264.dat.dec2 --hash-algorithm SHA-1 --mgf MGF1-SHA256

Using slot 0 with a present token (0x0)

Logging in to "YubiHSM".

Please enter User PIN: 

Using decrypt algorithm RSA-PKCS-OAEP

OAEP parameters: hashAlg=SHA-1, mgf=MGF1-SHA256, source_type=0, source_ptr=0x0, source_len=0

$ cmp t1264.dat t1264.dat.dec2

$

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mta.openssl.org/pipermail/openssl-dev/attachments/20170918/9ba4f774/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 5211 bytes
Desc: not available
URL: <http://mta.openssl.org/pipermail/openssl-dev/attachments/20170918/9ba4f774/attachment.bin>