Why do I get the following error `wrong signature length` when I try to validate a signed file using the c++ OpenSSL 3.1 library?

[Christian F. Gonzalez Di Antonio]

I posted this on
https://stackoverflow.com/questions/78604338/why-do-i-get-the-following-error-wrong-signature-length-when-i-try-to-validate

I'm writing an c++ program LicenseValidator ->
https://github.com/christiangda/LicenseValidator
<https://github.com/christiangda/LicenseValidator> to validate a
hypothetical program license using OpenSSL 3.1 Library
<https://wiki.openssl.org/index.php/OpenSSL_3.0>, and when I tried to
validate the licensed content I got the following error:

Failed to verify license
008C1AF901000000:error:02000077:rsa routines:ossl_rsa_verify:wrong
signature length:crypto/rsa/rsa_sign.c:338:
008C1AF901000000:error:1C880004:Provider routines:rsa_verify:RSA
lib:providers/implementations/signature/rsa_sig.c:785:

I would appreciate any help or guidance on what I am doing wrong.

I am not at all an expert in the c/c++ programming language and this is the
first time I have tried to use the OpenSSL library.

Of course, I've used GitHub Copilot, gemini, and chatgpt to write and
understand the repository code. The chalenge is about the examples I found
on internet, the majority of them are about OpenSSL v1 and the v3 is very
different, so was hard to understand the migration.

The README.md
<https://github.com/christiangda/LicenseValidator/blob/main/README.md> file
has the instructions to create all the necessary keys, etc, references I
used and the instructions to compile it using cmake.

The core function is LicenseValidator/src/License.cpp
<https://github.com/christiangda/LicenseValidator/blob/82f5501ab2e5bf2d91dc4298245b36fde2efd66b/src/License.cpp#L79>
:

bool verifyLicense(const unsigned char *licenseContent, const unsigned
char *licenseSignature, const std::string pubkey){
    EVP_PKEY *pkey = loadRsaPemPubKey(pubkey);
    if (pkey == NULL)
    {
        std::cerr << "Failed to load public key" << std::endl;
        ERR_print_errors_fp(stdout);
        return false;
    }

    EVP_PKEY_CTX *ctx = EVP_PKEY_CTX_new(pkey, NULL);
    if (ctx == NULL)
    {
        std::cerr << "Failed to create EVP_PKEY_CTX" << std::endl;
        EVP_PKEY_free(pkey);
        ERR_print_errors_fp(stdout);
        return false;
    }

    if (EVP_PKEY_verify_init(ctx) <= 0)
    {
        std::cerr << "Failed to initialize EVP_PKEY_CTX" << std::endl;
        EVP_PKEY_CTX_free(ctx);
        EVP_PKEY_free(pkey);
        ERR_print_errors_fp(stdout);
        return false;
    }

    // PKCS1 padding scheme
    if (EVP_PKEY_CTX_set_rsa_padding(ctx, RSA_PKCS1_PADDING) <= 0)
    {
        std::cerr << "Failed to set RSA padding" << std::endl;
        EVP_PKEY_CTX_free(ctx);
        EVP_PKEY_free(pkey);
        ERR_print_errors_fp(stdout);
        return false;
    }

    // SHA256 digest
    if (EVP_PKEY_CTX_set_signature_md(ctx, EVP_sha256()) <= 0)
    {
        std::cerr << "Failed to set signature MD" << std::endl;
        EVP_PKEY_CTX_free(ctx);
        EVP_PKEY_free(pkey);
        ERR_print_errors_fp(stdout);
        return false;
    }

    if (EVP_PKEY_verify(ctx, licenseSignature,
sizeof(licenseSignature), licenseContent, sizeof(licenseContent)) <=
0)
    {
        std::cerr << "Failed to verify license" << std::endl;
        EVP_PKEY_CTX_free(ctx);
        EVP_PKEY_free(pkey);
        ERR_print_errors_fp(stdout);
        return false;
    }

    EVP_PKEY_CTX_free(ctx);
    EVP_PKEY_free(pkey);

    return true;
}

Some guidance about how to solve the error I got.

-- 
Saludos,
Christian
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mta.openssl.org/pipermail/openssl-users/attachments/20240610/ad79d77b/attachment-0001.htm>