PKCS7_decrypt vs RSA OAEP padding

Tue Apr 13 14:07:23 UTC 2021

Hey Eliot,

Thank you for the PKCS7 vs CMS info.

# The test

you might try wrapping this stuff in PHP

I don't use PHP so I decided to do it C.
I was able to confirm my suspicion about PKCS7_decrypt not being able to decrypt message where RSA  is used with OAEP padding!
There are two programs attached (+ scep.h header file they both use):

- pkcs7.c is based on code from libscep library I mentioned.
- cms.c is very similar to pkcs7.c. I've only replaced the PKCS7 methods and data structures for CMS equivalents.

For them to work you need files named "scep.key", "scep.pem" and "pkimessage.p7b" present in current working directory <- I was lazy to use args.

My test files are also attached:

- pkimessage_rsa.p7b <- RSA encryption (Possibly with PKCS 1.5 padding?) is used. Both pkcs7.c and cms.c are able to decrypt the payload.
- pkimessage_rsaoaep.p7b <-  RSA encryption with OAEP padding is used. Only cms.c is able to decrypt the payload.
- scep.key and scep.pem (SCEP service certificate and private key I use for testing).

# How to fix my problem

There are multiple routes for me to solve this problem:

1. Convince SCEP client vendor (VMware) to let me configure encryption algorithm types or revert the change altogether (Still don't know if it was intentional).
2. Report this as a bug to OpenSSL and hope it gets fixed.
3. Replace the PKCS7_decrypt for CMS_decrypt in the libscep library.

## (1)

I have opened a ticket with their support week ago but it is _very_ hard to get hold of someone who has any idea what I am talking about.

## (2)

Back to my question from the original email.
 If it is indeed not capable of doing that would it be a bug or desired behavior?
Should I report this as a bug? If yes is there any chance this might be fixed soon (meaning in 1.1.1 branch instead of 3.0)?

## (3)

This is something I can do myself but there are two approaches.

A. Easy. Replace only PKCS7_decrypt method for CMS_decrypt method.
B. Hard. Rework the entire library to use CMS instead of PKCS7.

I would very like to go for the Easy choice here for two reasons. I am not able to do proper testing for such massive change.
For the same reason (plus one other) it is not very likely the big change would be accepted upstream

->
It there an easyway to convert PKCS7 struct into CMS_ContentInfo struct?
OR If there is not an easy way what would be a hard way to do it?

Michal Moravec

On 12. 4. 2021, at 19:24, Eliot Lear <lear at ofcourseimright.com<mailto:lear at ofcourseimright.com>> wrote:

Hi Michal,

CMS has limited backward compatibility with PKCS#7.  This is discussed in RFC 5652<https://tools.ietf.org/html/rfc5652#section-2>, and includes some suggestions as to how to some issues that might crop up.   At least the old draft of SCEP very specifically does NOT specify CMS, but that might not have stopped someone from using it.  These docs tend to be quite sloppy.  For example, I know one that uses both PEM and DER encodings on the wire.  Drives me batty.

For enrolment this is particularly embarrassing, considering its sensitivity.  As a sysadmin, you might try wrapping this stuff in PHP and just trying both the pkcs_decrypt and cms_decrypt calls and seeing if either work.

Eliot

On 12.04.21 14:13, Michal Moravec wrote:
Hello,

I am a system administrator trying to integrate two pieces of software using the SCEP protocol (more on that later in the More Context section) .
Integration was working but one side (SaaS product) changed the algorithms it uses for signing and encryption within the PKCS7 message.
After that I am unable to decrypt the PKCS7 messages on my end and get the error: "ERROR 139701710696896:error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt:../crypto/evp/evp_enc.c:597:
message_static_functions.c:221: decryption failed"

>From my current state of investigation it looks highly likely there is a problem with PKCS7_decrypt method which might be unable decrypt the PKCS7/CMS payload if the RSA is used together with OAEP padding.

1. What is a diffrences between pkcs7 https://github.com/openssl/openssl/tree/681618cfc18b4f01f2c07e823308d30f6f47504b/crypto/pkcs7 and CMS https://github.com/openssl/openssl/tree/681618cfc18b4f01f2c07e823308d30f6f47504b/crypto/cms implementations?

If I understand correctly CMS is just newer name for PKCS7 adopted by newer RFCs?
Do these two implementations overlap?
Or are there some kind of major diffrence like PKCS7 module being there for compatibility with older code and CMS is basically replacement to use for the future?

2. Is PKCS7_decrypt method in https://github.com/openssl/openssl/blob/681618cfc18b4f01f2c07e823308d30f6f47504b/crypto/pkcs7/pk7_smime.c capable of decrypting the PKCS7 payload whebf OAEP padding is used together with RSA to encrypt it? If it is indeed not capable of doing that would it be a bug or desired behavior?

If found a few pointers:

- https://stackoverflow.com/questions/56941480/how-to-set-padding-oaep-for-pkcs7-decrypt-function-using-openssl <- Developer is unable to use PKCS7_decrypt with RSA OAEP and forced to switch to CMS_decrypt method.
- https://stackoverflow.com/questions/34304570/how-to-resolve-the-evp-decryptfinal-ex-bad-decrypt-during-file-decryption <- Padding being a source problem of EVP_DecryptFinal_ex:bad decrypt error message.

3. How to replace PKCS7_decrypt method CMS_decrypt as easily as possible?

I have no hard proof of the PKCS7_decrypt being the culprit here so I started working on simple C program to reproduce the issue.
Problem is my C skills are very rusty and I have no experience working with OpenSSL C library.

int PKCS7_decrypt(PKCS7 *p7,                    EVP_PKEY *pkey, X509 *cert, BIO *data,                                  int flags);
int CMS_decrypt    (CMS_ContentInfo *cms, EVP_PKEY *pkey, X509 *cert, BIO *dcont, BIO *out, unsigned int flags);

Methods are very similiar but input data format it different. It there an easyway to convert PKCS7 struct into CMS_ContentInfo struct?

Original usage of PKCS7_decrypt here -> https://github.com/openxpki/libscep/blob/4067eae283ce0b3025d414e9d3b6af30def8c093/src/message_static_functions.c

# More Context

My goal is to obtain signed client certificate using the SCEP protocol.

SCEP server: OpenXPKI https://github.com/openxpki/openxpki (Perl) with libscep https://github.com/openxpki/libscep library (C). libscep used the OpenSSL.
SCEP client: VMware Workspace ONE UEM SaaS (WSO). Black box. From the error codes client produces I strongly suspect they use OpenSSL within .Net code.

Originally WSO SCEP client used these three algorithms for PKCS7 operation: SHA-1, RSA (PKCS 1.5 padding or no padding <- can't tell which) and 3DES-CBC.
They made some sort of change (unannouced) and since the last SaaS upgdate WSO client uses SHA-2 (256bit), RSA with OAEP padding and AES-256-CBC.
Compare the attached PKCS7 messages (libscep_wso*.p7b). I use https://lapo.it/asn1js to decode them easily.
After the change I am unable to decrypt the PKIOperation message on the server side.

I suspect the RSA with OAEP is the cause of the problem because using different client with SHA-256, AES-256-CBC and RSA decryption works without any problem (libscep_sscep_ok.p7b).

Best Regards,
Michal Moravec

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mta.openssl.org/pipermail/openssl-users/attachments/20210413/732dde6c/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: pkimessage_rsa.p7b
Type: application/octet-stream
Size: 3994 bytes
Desc: pkimessage_rsa.p7b
URL: <https://mta.openssl.org/pipermail/openssl-users/attachments/20210413/732dde6c/attachment-0006.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: pkimessage_rsaoaep.p7b
Type: application/octet-stream
Size: 4104 bytes
Desc: pkimessage_rsaoaep.p7b
URL: <https://mta.openssl.org/pipermail/openssl-users/attachments/20210413/732dde6c/attachment-0007.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: scep.key
Type: application/octet-stream
Size: 3294 bytes
Desc: scep.key
URL: <https://mta.openssl.org/pipermail/openssl-users/attachments/20210413/732dde6c/attachment-0008.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: scep.pem
Type: application/x-x509-ca-cert
Size: 1394 bytes
Desc: scep.pem
URL: <https://mta.openssl.org/pipermail/openssl-users/attachments/20210413/732dde6c/attachment-0001.crt>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: cms.c
Type: application/octet-stream
Size: 2759 bytes
Desc: cms.c
URL: <https://mta.openssl.org/pipermail/openssl-users/attachments/20210413/732dde6c/attachment-0009.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: scep.h
Type: application/octet-stream
Size: 9768 bytes
Desc: scep.h
URL: <https://mta.openssl.org/pipermail/openssl-users/attachments/20210413/732dde6c/attachment-0010.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: pkcs7.c
Type: application/octet-stream
Size: 2511 bytes
Desc: pkcs7.c
URL: <https://mta.openssl.org/pipermail/openssl-users/attachments/20210413/732dde6c/attachment-0011.obj>