How to use EVP_DigestSignInit[_ex]()?

[Alessandro Vesely]

Hi all,

my scarce crypto knowledge notwithstanding, I'm trying to maintain a DKIM 
signing function.  With the previous openssl version, I added a snippet to sign 
using ed25519, more or less like so:

     char *digest;              // digest already computed from
     size_t diglen;             // normalized message header

     EVP_PKEY *crypto_pkey;     // given

     size_t crypto_outlen;      // expected
     unsigned char *crypto_out;

     EVP_MD_CTX *md_ctx = EVP_MD_CTX_new();
     if (md_ctx == NULL) goto error;

     int status = EVP_DigestSignInit(md_ctx,
         NULL, NULL, NULL, crypto_pkey);
     if (status == 1)
         status = EVP_DigestSign(md_ctx,
             crypto_out, &crypto_outlen,
             digest, diglen);
     EVP_MD_CTX_free(md_ctx);
     if (status != 1) goto error;

That works.  With the onset of openssl 3 and RSA_sign() deprecation, I thought 
that snippet could work with RSA as well.  Given an RSA key, that code returns 
no error but a NULL crypto_out.  I should have specified SHA256 or (deprecated) 
SHA1, but I don't know how to obtain an EVP_MD *type.  Or is it better to use 
the _ex version?

Best
Ale
--