Support for retrieving hash algorithm from RSA signature before/during verification

[Atul Singh]

I found a solution to this problem. Just in case if someone finds this
later, I am using  EVP_PKEY_verify_recover, d2i_X509_SIG and
X509_SIG_get0  to retrieve the Hash algorithm from the signature.

My test code:

int
do_pk1_verifyrecover(EVP_PKEY * pkey, unsigned char * sig, size_t siglen,
                unsigned char * m, size_t mlen)
{
    int ret;
    EVP_PKEY_CTX * ctx;

    ctx = EVP_PKEY_CTX_new(pkey, NULL);
    EVP_PKEY_verify_recover_init(ctx);
    EVP_PKEY_CTX_set_rsa_padding(ctx, RSA_PKCS1_PADDING);
    ret = EVP_PKEY_verify_recover(ctx, m, &mlen, sig, siglen);

    // If the signature was not PKCS1 padded, an error should happen here
    if (ret <= 0) {
        printf("Sign failed with error %s\n",
               ERR_error_string(ERR_get_error(), NULL));
    } else {
        _print_buf("Recovered Message", m, mlen);
    }

    // Cleanup
    if (ctx)
        EVP_PKEY_CTX_free(ctx);

    return ret;
}

int
get_sig_algo_from_pkcs1_enc(const unsigned char * encoded_msg, size_t msglen)
{

    int ret;
    X509_SIG * sig;
    const X509_ALGOR * algo = NULL;
    const ASN1_OCTET_STRING * digest = NULL;

    sig = d2i_X509_SIG(NULL, &encoded_msg, msglen);
    X509_SIG_get0(sig, &algo, &digest);
    ret = OBJ_obj2nid(algo->algorithm);

    if (ret == NID_undef) {
        printf("Error getting algo NID from algo object");
        ret = -1;
    }

    _print_algo_name(algo);

    // Cleanup
    if (sig)
        X509_SIG_free(sig);

    return ret;
}

main()
...
    // 3. Recover message
    // After decryption, data will be == or < siglen
    // Allocate siglen bytes to be safe
    encoded_msg = malloc(siglen);
    msglen = siglen;
    if (do_pk1_verifyrecover(pkey, sig, siglen, encoded_msg, msglen) < 0){
    ret = -1;
    goto cleanup;
    }


    // 4. Retrieve Hash Algo from the recovered PKCS1 encoded message
    // Nvm about whether signature was pkcs1 or not because
    // if not previous setup would have generated an error.
    sig_algo_nid = get_sig_algo_from_pkcs1_enc(encoded_msg, msglen);
    md = EVP_get_digestbynid(sig_algo_nid);
    if (!md) {
    ret = -1;
    printf("Error: Unrecognized md in signature\n");
    goto cleanup;
    }


    // 5. Verify using the retrieved hash algo in previous step
    if (do_verify(pkey, md, sig, siglen) < 0){
    ret = -1;
    goto cleanup;

    }



On Mon, Apr 24, 2023 at 3:46 PM Atul Singh <singh.atulks at gmail.com> wrote:
>
> RSASSA-PKCS1-v1_5 signature scheme encodes the AlgorithmIdentifier of
> Hash Function in the message data before passing it through the
> encryption[1]. So, technically one doesn't need to know the Hash
> Function beforehand for verification purposes -- it can be decoded
> right out of the decrypted octets.
>
> Does openssl provide a way to perform verification without the
> application supplying the Hash Function for RSASSA-PKCS1-v1_5
> signatures? I have played around with EVP_DigestVerifyInit and
> friends, but it doesn't seem to be possible -- The Hash Function is
> required at the init context stage itself and there doesn't seem to be
> any way around it[2].
>
> I also did not see anything obvious in rsa.h that could just return
> the AlgorithmIdentifier that can be used for EVP_DigestVerifyInit
> later.
>
> The reason I am asking this question is that IKEv2 leaves it to the
> implementation to choose their Authentication method unannounced[3].
> If an IKE peer chooses RSA Signature as their authentication method,
> it can then go on to use any of the Hash Functions supported by RSA
> (RFC5996 only recommends making SHA1 default). This is causing a lot
> of incompatibility issues between vendors. Technically, one could try
> all possible algorithms one by one but that means doing 4 passes for
> SHA1, SHA256, SHA384 and SHA512 (assuming MD2 and MD5 are not in use
> anymore).
>
> Regards,
> Atul
>
> [1] https://datatracker.ietf.org/doc/html/rfc3447#section-9.2
> [2] https://www.openssl.org/docs/man1.1.1/man3/EVP_DigestInit.html
> [2] https://datatracker.ietf.org/doc/html/rfc5996#section-3.8