[openssl-users] Loading engines recursively and crypto engine lock

[Krzysztof Konopko]

Hi,

TL;DR;
Is it allowed to initialise engines recursively, ie. call `engine2->init`
from `engine1->init`?

--

I have a solution in a consumer product based on OpenSSL 1.0.2 series that
uses two engines: one (engine1) for selecting client certificate chain (TLS
client auth) and another one (engine2) for RPC operations on associated
private keys stored in H/W.  This works only if supplied (installed) locks
are recursive as for each engine initialisation `CRYPTO_LOCK_ENGINE` is
taken.

>From what I see, OpenSSL 1.1.x onward, provides locking internally and it's
non-recursive.  Also `lock_dbg_cb()` implementation in OpenSSL before 1.1.x
suggests locks are not expected to be recursive.

Here's some more context of my use case.​

OpenSSL loads `engine1` for me automatically (`
​​
​
OPENSSL_SSL_CLIENT_ENGINE_AUTO` variable) which is convenient as I don't
have control over application's `main()` function.  In my case it's
proprietary code but equally it could be Python script (I do not fancy
patching Python interpreter to get to its `main()` function and
load/initialise engines explicitly).

So my _only_ entry point is `engine1->init`.  In that entry point I
initialise engine2 which is a fairly slow operation (need to load certs
from permanent storage) so definitely want to do this only once. Oh, and
the app is heavily multi-threaded so I'm glad OpenSSL carefully takes
crypto engine lock where needed.

But because engines are initialised recursively, the locking implementation
I supply uses recursive mutex which works very well and makes perfect sense
to me in this case (I know that the same thread calls locked functions
recursively for a reason).  This works only before 1.1.x.

Alternatively I could lazy-initialise engine2 in certificate callback
function but any initialisation failure here would be less meaningful and
it would require another lock to protect engine2 handle.  In
`engine1->init` I know a lock is already held so I thought it's safer to do
more initialisation here.  Besides `engine2->init` is not called directly
but through a layer of application logic so conceptually these two engines
are orthogonal and know nothing about each other.

I guess initialising engines recursively does not work in OpenSSL 1.1.x
(it'd be a dead-lock) and I need to seek for a different place to
initialise engine2, for example in certificate cb?  This would mean I
"leak" some knowledge of engine2 existence into engine1, have guarantee
that crypto engine lock is not held in certificate callabck function and
need another lock to protect access to engine2 handle.

Please let me know what your views are and if the above makes sense.

Thanks,
Kris
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mta.openssl.org/pipermail/openssl-users/attachments/20160808/d4521d42/attachment.html>