[openssl-users] How to plug in different digest algorithm implementation into the PKCS7 functions?

[Stephan Mühlstrasser]

Hi,

I'm trying to plug my own digest algorithm implementation into the PKCS7 
functions for creating a signature (using OpenSSL 1.0.2). The hash 
computation shall be performed on a hardware device.

For that purpose I wanted to supply my own EVP_MD data structure to 
PKCS7_add_signature(). A rough sketch of my code for replacing the 
standard SHA-256 implementation looks like this:

     static const EVP_MD my_digest_impl =
     {
         NID_sha256,
         ...
         /* contains function pointers for my own implementation */
     };

     PKCS7 *p7 = PKCS7_new();

     PKCS7_set_type(p7, NID_pkcs7_signed);

     PKCS7_SIGNER_INFO *si = PKCS7_add_signature(p7, cert, pkey, 
&my_digest_impl);

     PKCS7_content_new(sig_parms->p7, NID_pkcs7_data);

     PKCS7_set_detached(p7, 1);

     BIO *p7bio = PKCS7_dataInit(p7, NULL);
     ...

When I debug through this code, I can see that OpenSSL does not call the 
"init" function pointer of the "my_digest_impl" structure, but it calls 
OpenSSL's standard SHA-256 init function "init256". The stack looks like 
this:

init256() at .../openssl-src/crypto/evp/m_sha1.c:107	
EVP_DigestInit_ex() at .../openssl-src/crypto/evp/digest.c:256	
md_ctrl() at .../openssl-src/crypto/evp/bio_md.c:220	
BIO_ctrl() at .../openssl-src/crypto/bio/bio_lib.c:370	
PKCS7_bio_add_digest() at .../openssl-src/crypto/pkcs7/pk7_doit.c:122	
PKCS7_dataInit() at .../openssl-src/crypto/pkcs7/pk7_doit.c:319	
...

How can I plug in my own digest implementation? Do I need to implement a 
full OpenSSL engine for this purpose?

Thanks

--
Stephan