Question on porting custom ENGINE to provider (OpenSSL v3.0.10)

[Timo Herbrecher]

Hello together.

I've a hardware secure element (SE) that only supports the following 
functionalities:
- hold EC key pair
- provide EC public key and curve type on request
- sign a pre-hashed (SHA1, SHA256, SHA384 or SHA512) data block with the 
private EC key

I've implemented a custom provider to interact with this SE with the 
following functions:
- store management
- key management (only get key from storage and export it)
- signing (only signature_digest_sign related functions)
Before switching to OpenSSL v3.0.10 everything was handled by a ENGINE 
implementation which worked great.

Everything is working fine if I interact with the provider through 
openssl CLI:
- openssl ec -provider /usr/lib/libcustom_key_provider -provider default 
-in keystore:1 -pubout
- openssl dgst -provider /usr/lib/libcustom_key_provider.so -provider 
default -sign keystore:0 -sha256 -out /root/tbs.sign /root/tbs

The algorithm name of key management and signing are both set to 'EC' so 
e.g. the default formatter could be used to print out the public key.

But when I try to use the provider in my TLS server implementation it 
does not work as expected and I am lost here.

I load the pkey from the provider via OSSL_STORE_open -> OSSL_STORE_load 
-> OSSL_STORE_INFO_get1_pkey and bind it to the certificate related to 
the key pair with SSL_CTX_use_PrivateKey in the SSL_CTX. And the 
ServerHello and ServerCertificate messages are created. So far so good.

But the ServerKeyExchange is not generated and the TLS handshake aborts.

I assume the problem here is that my provider is loaded as the provider 
for all EC algorithm related functions. But I need the default provider 
to handle the key exchange to generate and derive the transport key.

Is it somehow possible to just use my provider as intended for digest 
signing of stuff related to my server certificate? Or do I have to 
re-invent the wheel on my provider and also implement key generation, 
key exchange and ciphers even if the SE does not support anything of that?

I've read about the provider properties (propquery) that could be used 
for provider selection but I don't understand how to use them to reach 
my goal.

Maybe someone could point me in the right direction? I'm stuck here for 
over a week now trying and debugging different things.


Thanks in advance and best regards,

Timo
-------------- next part --------------
A non-text attachment was scrubbed...
Name: OpenPGP_signature.asc
Type: application/pgp-signature
Size: 840 bytes
Desc: OpenPGP digital signature
URL: <https://mta.openssl.org/pipermail/openssl-users/attachments/20231117/ef05cd2a/attachment-0001.sig>