Updating RSA public key generation and signature verification from 1.1.1 to 3.0

[Michael Wojcik]

> From: openssl-users <openssl-users-bounces at openssl.org> On Behalf Of Tomas
> Mraz
> Sent: Friday, 30 September, 2022 00:22
> 
> unfortunately I do not see anything wrong with the code. Does the
> EVP_DigestVerifyFinal return 0 or negative value? I do not think this
> is a bug in OpenSSL as this API is thoroughly tested and it is highly
> improbable that there would be a bug in the ECDSA verification through
> this API.
> 
> I am currently out of ideas on what could be wrong or how to
> investigate further. Perhaps someone else can chime in on what can be
> wrong?

Coincidentally, just yesterday I was helping someone debug a DigestVerify issue. We were consistently getting the "first octet is invalid" error out of the RSA PSS signature verification code, but the same inputs worked with openssl dgst.

I wrote a fresh minimal program from scratch (really minimal, with hard-coded filenames for the inputs), and it worked fine as soon as it compiled cleanly.

I'd suggest trying that. Get it working in a minimal program first. Make sure you have all the correct OpenSSL headers, and there are no compilation warnings. Then integrate that code into your application.

(I didn't have the original application to go back to, in my case, and the person I was working with is in another timezone and had left for the day.)

-- 
Michael Wojcik
Distinguished Engineer, Application Modernization and Connectivity