Howto prevent cycles in engine invocation ?

[Fuchs, Andreas]

Hi Dimitry,

thanks, yes, that's why I'm doing inside the engine as well.

The question was regarding the library that my engine links against that in turn uses libcrypto again.

That linked library uses the EVP methods with the "default"-engine, which is however set from the surrounding application.

I guess it's a similar question to when an application links against liba.so and libb.so and both use openssl internally but set different default engines. How can you prevent them from interfering with each other ?

Thanks,
Andreas

________________________________
From: openssl-users [openssl-users-bounces at openssl.org] on behalf of Dmitry Belyavsky [beldmit at gmail.com]
Sent: Tuesday, March 19, 2019 21:09
To: openssl-users at openssl.org
Subject: Re: Howto prevent cycles in engine invocation ?

Hello Andreas,

I used smth like
=======
RSA_METHOD my_rsa_method = {
    "My RSA method",
    0,                          /* pub_enc */
    0,                          /* pub_dec */
    0,          /* priv_enc */
    my_priv_dec,          /* priv_dec */
    0,                          /* rsa_mod_exp */
    0,                          /* bn_mod_exp */
    0,                          /* init */
    0,              /* finish */
    RSA_METHOD_FLAG_NO_CHECK|RSA_FLAG_SIGN_VER,          /* flags */
    NULL,                       /* app_data */
    my_rsa_sign,              /* rsa_sign */
    0                           /* rsa_verify */
};

static int my_priv_dec (int flen, const unsigned char *from,
    unsigned char *to, RSA *rsa, int padding)
{
  const RSA_METHOD *def_meth = RSA_PKCS1_SSLeay();
  if ((rsa->meth == &my_rsa_method) && RSA_get_ex_data(rsa, my_key_pos))
  {
    return my_op_rsa_decrypt(flen, from, to, rsa, padding);
  }

  return def_meth->rsa_priv_dec(flen, from, to, rsa, padding);
}
==============
But this code worked for 1.0.* branch

вт, 19 марта 2019 г., 19:52 Fuchs, Andreas <andreas.fuchs at sit.fraunhofer.de<mailto:andreas.fuchs at sit.fraunhofer.de>>:
Following scenario:

I have an engine that implements e.g. RSA decryption.
That engine links against a library that links against libcrypto to perform RSA decryption.

Now if I have an application that sets the default library to be my engine, won't I end up in an infinite loop ?

Or the other way around:
- Is there a way to change the default engine from within my engine before calling out to the library ?
- Can the library itself select "software-only" as an engine ?

I guess I don't have a concrete problem in my specific case, since I have app_data attached to any key and
if my engine receives a key without app_data, it will just call SSLeay() functions.

But what if that was not the case ?

Thanks a lot for any help,
Andreas
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mta.openssl.org/pipermail/openssl-users/attachments/20190319/7ee9f565/attachment.html>