PKCS7_decrypt vs RSA OAEP padding

[Eliot Lear]

Hi Michal,

CMS has limited backward compatibility with PKCS#7.  This is discussed 
in RFC 5652 <https://tools.ietf.org/html/rfc5652#section-2>, and 
includes some suggestions as to how to some issues that might crop up.   
At least the old draft of SCEP very specifically does NOT specify CMS, 
but that might not have stopped someone from using it.  These docs tend 
to be quite sloppy.  For example, I know one that uses both PEM and DER 
encodings on the wire.  Drives me batty.

For enrolment this is particularly embarrassing, considering its 
sensitivity.  As a sysadmin, you might try wrapping this stuff in PHP 
and just trying both the pkcs_decrypt and cms_decrypt calls and seeing 
if either work.

Eliot

On 12.04.21 14:13, Michal Moravec wrote:
> Hello,
>
> I am a system administrator trying to integrate two pieces of software 
> using the SCEP protocol (more on that later in the More Context section) .
> Integration was working but one side (SaaS product) changed the 
> algorithms it uses for signing and encryption within the PKCS7 message.
> After that I am unable to decrypt the PKCS7 messages on my end and get 
> the error: "ERROR 139701710696896:error:06065064:digital envelope 
> routines:EVP_DecryptFinal_ex:bad decrypt:../crypto/evp/evp_enc.c:597:
> message_static_functions.c:221: decryption failed"
>
> From my current state of investigation it looks highly likely there is 
> a problem with PKCS7_decrypt method which might be unable decrypt the 
> PKCS7/CMS payload if the RSA is used together with OAEP padding.
>
> 1. What is a diffrences between pkcs7 
> https://github.com/openssl/openssl/tree/681618cfc18b4f01f2c07e823308d30f6f47504b/crypto/pkcs7 
> <https://github.com/openssl/openssl/tree/681618cfc18b4f01f2c07e823308d30f6f47504b/crypto/pkcs7> and 
> CMS 
> https://github.com/openssl/openssl/tree/681618cfc18b4f01f2c07e823308d30f6f47504b/crypto/cms 
> <https://github.com/openssl/openssl/tree/681618cfc18b4f01f2c07e823308d30f6f47504b/crypto/cms> implementations?
>
> If I understand correctly CMS is just newer name for PKCS7 adopted by 
> newer RFCs?
> Do these two implementations overlap?
> Or are there some kind of major diffrence like PKCS7 module being 
> there for compatibility with older code and CMS is basically 
> replacement to use for the future?
>
> 2. Is PKCS7_decrypt method in 
> https://github.com/openssl/openssl/blob/681618cfc18b4f01f2c07e823308d30f6f47504b/crypto/pkcs7/pk7_smime.c 
> <https://github.com/openssl/openssl/blob/681618cfc18b4f01f2c07e823308d30f6f47504b/crypto/pkcs7/pk7_smime.c> capable 
> of decrypting the PKCS7 payload whebf OAEP padding is used together 
> with RSA to encrypt it? If it is indeed not capable of doing that 
> would it be a bug or desired behavior?
>
> If found a few pointers:
>
> - 
> https://stackoverflow.com/questions/56941480/how-to-set-padding-oaep-for-pkcs7-decrypt-function-using-openssl 
> <https://stackoverflow.com/questions/56941480/how-to-set-padding-oaep-for-pkcs7-decrypt-function-using-openssl> <- 
> Developer is unable to use PKCS7_decrypt with RSA OAEP and forced to 
> switch to CMS_decrypt method.
> - 
> https://stackoverflow.com/questions/34304570/how-to-resolve-the-evp-decryptfinal-ex-bad-decrypt-during-file-decryption 
> <https://stackoverflow.com/questions/34304570/how-to-resolve-the-evp-decryptfinal-ex-bad-decrypt-during-file-decryption> <- 
> Padding being a source problem of EVP_DecryptFinal_ex:bad decrypt 
> error message.
>
> 3. How to replace PKCS7_decrypt method CMS_decrypt as easily as possible?
>
> I have no hard proof of the PKCS7_decrypt being the culprit here so I 
> started working on simple C program to reproduce the issue.
> Problem is my C skills are very rusty and I have no experience working 
> with OpenSSL C library.
>
>> int PKCS7_decrypt(PKCS7 *p7,  EVP_PKEY *pkey, X509 *cert, BIO *data, 
>>              int flags);
>> int CMS_decrypt    (CMS_ContentInfo *cms, EVP_PKEY *pkey, X509 *cert, 
>> BIO *dcont, BIO *out, unsigned int flags);
>
> Methods are very similiar but input data format it different. It there 
> an easyway to convert PKCS7 struct into CMS_ContentInfo struct?
>
> Original usage of PKCS7_decrypt here -> 
> https://github.com/openxpki/libscep/blob/4067eae283ce0b3025d414e9d3b6af30def8c093/src/message_static_functions.c 
> <https://github.com/openxpki/libscep/blob/4067eae283ce0b3025d414e9d3b6af30def8c093/src/message_static_functions.c> 
>
>
>
> # More Context
>
> My goal is to obtain signed client certificate using the SCEP protocol.
>
> SCEP server: OpenXPKI https://github.com/openxpki/openxpki 
> <https://github.com/openxpki/openxpki> (Perl) with libscep 
> https://github.com/openxpki/libscep 
> <https://github.com/openxpki/libscep> library (C). libscep used the 
> OpenSSL.
> SCEP client: VMware Workspace ONE UEM SaaS (WSO). Black box. From the 
> error codes client produces I strongly suspect they use OpenSSL within 
> .Net code.
>
> Originally WSO SCEP client used these three algorithms for PKCS7 
> operation: SHA-1, RSA (PKCS 1.5 padding or no padding <- can't tell 
> which) and 3DES-CBC.
> They made some sort of change (unannouced) and since the last SaaS 
> upgdate WSO client uses SHA-2 (256bit), RSA with OAEP padding and 
> AES-256-CBC.
> Compare the attached PKCS7 messages (libscep_wso*.p7b). I use 
> https://lapo.it/asn1js <https://lapo.it/asn1js> to decode them easily.
> After the change I am unable to decrypt the PKIOperation message on 
> the server side.
>
> I suspect the RSA with OAEP is the cause of the problem because using 
> different client with SHA-256, AES-256-CBC and RSA decryption works 
> without any problem (libscep_sscep_ok.p7b).
>
> Best Regards,
> Michal Moravec
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mta.openssl.org/pipermail/openssl-users/attachments/20210412/6f6b42ca/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: OpenPGP_signature
Type: application/pgp-signature
Size: 495 bytes
Desc: OpenPGP digital signature
URL: <https://mta.openssl.org/pipermail/openssl-users/attachments/20210412/6f6b42ca/attachment-0001.sig>