Howto prevent cycles in engine invocation ?

Fuchs, Andreas andreas.fuchs at
Tue Mar 19 22:31:21 UTC 2019

Hi Dimitry,

thanks, yes, that's why I'm doing inside the engine as well.

The question was regarding the library that my engine links against that in turn uses libcrypto again.

That linked library uses the EVP methods with the "default"-engine, which is however set from the surrounding application.

I guess it's a similar question to when an application links against and and both use openssl internally but set different default engines. How can you prevent them from interfering with each other ?


From: openssl-users [openssl-users-bounces at] on behalf of Dmitry Belyavsky [beldmit at]
Sent: Tuesday, March 19, 2019 21:09
To: openssl-users at
Subject: Re: Howto prevent cycles in engine invocation ?

Hello Andreas,

I used smth like
RSA_METHOD my_rsa_method = {
    "My RSA method",
    0,                          /* pub_enc */
    0,                          /* pub_dec */
    0,          /* priv_enc */
    my_priv_dec,          /* priv_dec */
    0,                          /* rsa_mod_exp */
    0,                          /* bn_mod_exp */
    0,                          /* init */
    0,              /* finish */
    RSA_METHOD_FLAG_NO_CHECK|RSA_FLAG_SIGN_VER,          /* flags */
    NULL,                       /* app_data */
    my_rsa_sign,              /* rsa_sign */
    0                           /* rsa_verify */

static int my_priv_dec (int flen, const unsigned char *from,
    unsigned char *to, RSA *rsa, int padding)
  const RSA_METHOD *def_meth = RSA_PKCS1_SSLeay();
  if ((rsa->meth == &my_rsa_method) && RSA_get_ex_data(rsa, my_key_pos))
    return my_op_rsa_decrypt(flen, from, to, rsa, padding);

  return def_meth->rsa_priv_dec(flen, from, to, rsa, padding);
But this code worked for 1.0.* branch

вт, 19 марта 2019 г., 19:52 Fuchs, Andreas <andreas.fuchs at<mailto:andreas.fuchs at>>:
Following scenario:

I have an engine that implements e.g. RSA decryption.
That engine links against a library that links against libcrypto to perform RSA decryption.

Now if I have an application that sets the default library to be my engine, won't I end up in an infinite loop ?

Or the other way around:
- Is there a way to change the default engine from within my engine before calling out to the library ?
- Can the library itself select "software-only" as an engine ?

I guess I don't have a concrete problem in my specific case, since I have app_data attached to any key and
if my engine receives a key without app_data, it will just call SSLeay() functions.

But what if that was not the case ?

Thanks a lot for any help,
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <>

More information about the openssl-users mailing list