Updating RSA public key generation and signature verification from 1.1.1 to 3.0

[Tomas Mraz]

On Thu, 2022-09-08 at 16:10 +0000, GonzalezVillalobos, Diego via
openssl-users wrote:
> [AMD Official Use Only - General]
> 
> Hello everyone,
>  
> I am currently working on updating a signature verification function
> in C++ and I am a bit stuck. I am trying to replace the deprecated
> 1.1.1 functions to the appropriate 3.0 versions. The function takes
> in 2 certificate objects (parent and cert), which are not x509
> certificates, but certificates the company had previously defined.
> Using the contents from parent we create an RSA public key and using
> the contents from cert we create the digest and grab the signature to
> verify.
>  
> In the 1.1.1 version we were using the RSA Object and the
> rsa_set0_key function to create the RSA public key and then used
> RSA_public_decrypt to decrypt the signature and RSA_verify_PKCS1_PSS
> to verify it. This whole workflow is now deprecated.
> 
...
> Is this the correct way of creating RSA keys now? Where is my logic
> failing? Can the same type of procedure even be done on 3.0? Any
> advice would be really appreciated.
>  

In the original code you seem to be using PSS padding for verification.
Did you try to set the PSS padding on the digest verify context? See
demos/signature/rsa_pss_hash.c on how to do it.

-- 
Tomáš Mráz, OpenSSL