[openssl-dev] Bug: digest parameter is rejected

[Blumenthal, Uri - 0553 - MITLL]

RSA-OAEP supports different hash functions and MGF. SHA-1 is the default.

 

OpenSSL implementation of OAEP wrongly refuses to set the hash algorithm, preventing one from using SHA-2 family:

 

$ openssl version

OpenSSL 1.0.2l  25 May 2017

$ openssl pkeyutl -encrypt -in t1264.dat -out t1264.dat.enc2.oaep -keyform DER -pubin -inkey rsa3072pub.der -pkeyopt rsa_padding_mode:oaep

$ openssl pkeyutl -encrypt -in t1264.dat -out t1264.dat.enc2.oaep -keyform DER -pubin -inkey rsa3072pub.der -pkeyopt rsa_padding_mode:oaep -pkeyopt digest:sha256

parameter setting error

140736155067400:error:06089094:digital envelope routines:EVP_PKEY_CTX_ctrl:invalid operation:pmeth_lib.c:376:

$ ~/openssl-1.1/bin/openssl version

OpenSSL 1.1.0g-dev  xx XXX xxxx

$ ~/openssl-1.1/bin/openssl pkeyutl -encrypt -in t1264.dat -out t1264.dat.enc2.oaep -keyform DER -pubin -inkey rsa3072pub.der -pkeyopt rsa_padding_mode:oaep

$ ~/openssl-1.1/bin/openssl pkeyutl -encrypt -in t1264.dat -out t1264.dat.enc2.oaep -keyform DER -pubin -inkey rsa3072pub.der -pkeyopt rsa_padding_mode:oaep -pkeyopt digest:sha256

pkeyutl: Can't set parameter:

140736155067328:error:06089094:digital envelope routines:EVP_PKEY_CTX_ctrl:invalid operation:crypto/evp/pmeth_lib.c:312:

$

 

It seems that OpenSSL tries to enforce the incorrect assumption that digest/hash is only applicable to signature padding, but not to encryption padding?

--

Regards,

Uri Blumenthal

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mta.openssl.org/pipermail/openssl-dev/attachments/20170918/23ad2b95/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 5211 bytes
Desc: not available
URL: <http://mta.openssl.org/pipermail/openssl-dev/attachments/20170918/23ad2b95/attachment.bin>