PKCS7_decrypt vs RSA OAEP padding

Mon Apr 12 12:13:22 UTC 2021

Hello,

I am a system administrator trying to integrate two pieces of software using the SCEP protocol (more on that later in the More Context section) .
Integration was working but one side (SaaS product) changed the algorithms it uses for signing and encryption within the PKCS7 message.
After that I am unable to decrypt the PKCS7 messages on my end and get the error: "ERROR 139701710696896:error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt:../crypto/evp/evp_enc.c:597:
message_static_functions.c:221: decryption failed"

>From my current state of investigation it looks highly likely there is a problem with PKCS7_decrypt method which might be unable decrypt the PKCS7/CMS payload if the RSA is used together with OAEP padding.

1. What is a diffrences between pkcs7 https://github.com/openssl/openssl/tree/681618cfc18b4f01f2c07e823308d30f6f47504b/crypto/pkcs7 and CMS https://github.com/openssl/openssl/tree/681618cfc18b4f01f2c07e823308d30f6f47504b/crypto/cms implementations?

If I understand correctly CMS is just newer name for PKCS7 adopted by newer RFCs?
Do these two implementations overlap?
Or are there some kind of major diffrence like PKCS7 module being there for compatibility with older code and CMS is basically replacement to use for the future?

2. Is PKCS7_decrypt method in https://github.com/openssl/openssl/blob/681618cfc18b4f01f2c07e823308d30f6f47504b/crypto/pkcs7/pk7_smime.c capable of decrypting the PKCS7 payload whebf OAEP padding is used together with RSA to encrypt it? If it is indeed not capable of doing that would it be a bug or desired behavior?

If found a few pointers:

- https://stackoverflow.com/questions/56941480/how-to-set-padding-oaep-for-pkcs7-decrypt-function-using-openssl <- Developer is unable to use PKCS7_decrypt with RSA OAEP and forced to switch to CMS_decrypt method.
- https://stackoverflow.com/questions/34304570/how-to-resolve-the-evp-decryptfinal-ex-bad-decrypt-during-file-decryption <- Padding being a source problem of EVP_DecryptFinal_ex:bad decrypt error message.

3. How to replace PKCS7_decrypt method CMS_decrypt as easily as possible?

I have no hard proof of the PKCS7_decrypt being the culprit here so I started working on simple C program to reproduce the issue.
Problem is my C skills are very rusty and I have no experience working with OpenSSL C library.

int PKCS7_decrypt(PKCS7 *p7,                    EVP_PKEY *pkey, X509 *cert, BIO *data,                                  int flags);
int CMS_decrypt    (CMS_ContentInfo *cms, EVP_PKEY *pkey, X509 *cert, BIO *dcont, BIO *out, unsigned int flags);

Methods are very similiar but input data format it different. It there an easyway to convert PKCS7 struct into CMS_ContentInfo struct?

Original usage of PKCS7_decrypt here -> https://github.com/openxpki/libscep/blob/4067eae283ce0b3025d414e9d3b6af30def8c093/src/message_static_functions.c

# More Context

My goal is to obtain signed client certificate using the SCEP protocol.

SCEP server: OpenXPKI https://github.com/openxpki/openxpki (Perl) with libscep https://github.com/openxpki/libscep library (C). libscep used the OpenSSL.
SCEP client: VMware Workspace ONE UEM SaaS (WSO). Black box. From the error codes client produces I strongly suspect they use OpenSSL within .Net code.

Originally WSO SCEP client used these three algorithms for PKCS7 operation: SHA-1, RSA (PKCS 1.5 padding or no padding <- can't tell which) and 3DES-CBC.
They made some sort of change (unannouced) and since the last SaaS upgdate WSO client uses SHA-2 (256bit), RSA with OAEP padding and AES-256-CBC.
Compare the attached PKCS7 messages (libscep_wso*.p7b). I use https://lapo.it/asn1js to decode them easily.
After the change I am unable to decrypt the PKIOperation message on the server side.

I suspect the RSA with OAEP is the cause of the problem because using different client with SHA-256, AES-256-CBC and RSA decryption works without any problem (libscep_sscep_ok.p7b).

Best Regards,
Michal Moravec

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mta.openssl.org/pipermail/openssl-users/attachments/20210412/c01f15fe/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: libscep_wso_after_broken.p7b
Type: application/octet-stream
Size: 4104 bytes
Desc: libscep_wso_after_broken.p7b
URL: <https://mta.openssl.org/pipermail/openssl-users/attachments/20210412/c01f15fe/attachment-0003.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: libscep_wso_before_ok.p7b
Type: application/octet-stream
Size: 4124 bytes
Desc: libscep_wso_before_ok.p7b
URL: <https://mta.openssl.org/pipermail/openssl-users/attachments/20210412/c01f15fe/attachment-0004.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: libscep_sscep_ok.p7b
Type: application/octet-stream
Size: 3950 bytes
Desc: libscep_sscep_ok.p7b
URL: <https://mta.openssl.org/pipermail/openssl-users/attachments/20210412/c01f15fe/attachment-0005.obj>