[openssl-dev] EC Keys with a statically linked engine

[Matt Hauck]

>From my testing, it appears that EC keys that are managed by an engine (in
my case, libp11 <https://github.com/OpenSC/libp11>) are not presently
supported by OpenSSL when the engine is statically linked. This appears
primarily to be due to the implementation of the `ecdsa_check` function in
how it looks up the right method data to call.

(Context: I am currently writing a library that statically links OpenSSL
and am trying to load libp11, statically linked against openssl too. I do
not want to link libp11 statically due to licensing concerns. I have tested
with OpenSSL 1.0.1t and 1.0.2h)

Basically, when I call ECDSA_do_sign from my wrapper library, it calls
`ecdsa_check` with function pointers to its own copies of `ecdsa_free`,
`ecdsa_dup`. However, when the libp11 library loaded the private key and
set its method data (with ECDSA_set_method) from its context, it has
function pointers to its own copies of `ecdsa_free` and friends, and thus
the `EC_KEY_get_key_method_data` function always returns NULL, resulting in
openssl using its own routines to sign/decrypt/etc. rather than using the
engine's routines.

It seems to me that engines which statically link openssl will never work
with EC keys.

I have tested around with RSA keys as well, and they work fine as they do
not have this `ecdsa_check` like function that filters based on function
pointers. The method data there appears to be stored directly on the
private key object.

Questions:
1. Why is method data stored differently on EC keys than on RSA keys?
2. Was there a reason to do this filtering with function pointers this way?
Was it only to distinguish ECDSA and ECDH function data?
3. If we were to patch openssl to store this method data directly on the EC
key struct, is there anything we should be aware of breaking? Would a patch
along these lines be accepted upstream?

Thank you!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mta.openssl.org/pipermail/openssl-dev/attachments/20160712/3df09b13/attachment.html>