CMS decryption of message with OAEP using Hardware security module

[Thulasi Goriparthi]

Do you mean RSA OAEP decryption done by HSM fails?

Possible tests:
1. Try RSA OAEP encryption/decryption with HSM. - basic test.
2. Encrypt with HSM and decrypt using openssl crypto library. - To make
sure RSA OAEP encryption of the HSM works fine.
3. If test 2 fails, check if all the parameters (hash, mgf, salt length)
used for OAEP are same on both sides. If they match and decryption still
fails, check with your HSM vendor. If they don't, try fixing the parameters
and repeat test 2.

RSA_NO_PADDING always works as all it does is modular exponentiation.

Thanks,
Thulasi.

On Mon, 17 Feb, 2020, 19:22 RudyAC, <rpo at compumatica.com> wrote:

> Hi,
>
> I have the requirement to decrypt e-mails where RSA-OAEP padding is used. I
> use the library openssl-1.0.2k and decrypt with CMS container
> (CMS_decrypt).
> This works very well unless the private key is stored in a Hardware
> security
> module and the cryptographic operation is performed via the PKCS11 engine
> from openssl.
>
> When decrypting an email which uses OAEP I got the error message:
>
> 47235129370352:error:06065064:digital envelope
> routines:EVP_DecryptFinal_ex:bad decrypt:evp_enc.c:529:
>
> To analyze the problem I encrypted an clear text using OAEP padding and
> setup a decryption function using
> RSA_private_decrypt(). Here I use padding mode "RSA_NO_PADDING" and the
> decryption also works with the PKCS11 engine. Unfortunately CMS does not
> support setting the padding mode.
>
> For any comments I would be very grateful
>
> Regards Rudy
>
>
>
> --
> Sent from: http://openssl.6102.n7.nabble.com/OpenSSL-User-f3.html
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mta.openssl.org/pipermail/openssl-users/attachments/20200218/482863c5/attachment.html>