[openssl-users] Question on implementing the ameth ctrl ASN1_PKEY_CTRL_DEFAULT_MD_NID

[Fuchs, Andreas]

Hi all,

I'm currently implementing a TPM2 engine for OpenSSL over at https://github.com/tpm2-software/tpm2-tss-engine
The problem I'm facing is that OpenSSL's TLS negotiation will request ECDSA from my engine with any hash alg, even though the TPM's keys are restricted to just one specific hash alg.

Most recently, David Woodhouse pointed out the possibility to require a certain hash-alg from the key to TLS via the ameth ASN1_PKEY_CTRL_DEFAULT_MD_NID at https://github.com/tpm2-software/tpm2-tss-engine/issues/31

Since I'm not that familiar with OpenSSL, I wanted to confirm that I'm following the right path for implementing this.
Thus: Is the following approach correct ?

So, at https://github.com/tpm2-software/tpm2-tss-engine/blob/master/src/tpm2-tss-engine-ecc.c#L328:
- I need to call "const EVP_PKEY_ASN1_METHOD *EVP_PKEY_get0_asn1(const EVP_PKEY *pkey)" to get the ameth ?
- I need to call EVP_PKEY_asn1_set_ctrl(EVP_PKEY_ASN1_METHOD *ameth, (*pkey_ctrl)) to some pkey_ctrl for ECC keys of mine ?
- That pkey_ctrl is a int (*pkey_ctrl) (EVP_PKEY *pkey, int op, long arg1, void *arg2)) that implements the op ASN1_PKEY_CTRL_DEFAULT_MD_NID ?
- That pkey_ctrl()'s ASN1_PKEY_CTRL_DEFAULT_MD_NID looks up the hash for the provided pkey's ecc key from the tpm2data and returns it via *(int *)arg2 = NID_sha1 or NID_sha256 or etc and then returns 1 or 2 or something ?
- Which one of the return codes (1 or 2) makes it mandatory rather than recommended ?

Thanks a lot for any advice,
Andreas