Howto prevent cycles in engine invocation ?

[Dmitry Belyavsky]

Hello Andreas,

I used smth like
=======
RSA_METHOD my_rsa_method = {
    "My RSA method",
    0,                          /* pub_enc */
    0,                          /* pub_dec */
    0,          /* priv_enc */
    my_priv_dec,          /* priv_dec */
    0,                          /* rsa_mod_exp */
    0,                          /* bn_mod_exp */
    0,                          /* init */
    0,              /* finish */
    RSA_METHOD_FLAG_NO_CHECK|RSA_FLAG_SIGN_VER,          /* flags */
    NULL,                       /* app_data */
    my_rsa_sign,              /* rsa_sign */
    0                           /* rsa_verify */
};

static int my_priv_dec (int flen, const unsigned char *from,
    unsigned char *to, RSA *rsa, int padding)
{
  const RSA_METHOD *def_meth = RSA_PKCS1_SSLeay();
  if ((rsa->meth == &my_rsa_method) && RSA_get_ex_data(rsa, my_key_pos))
  {
    return my_op_rsa_decrypt(flen, from, to, rsa, padding);
  }

  return def_meth->rsa_priv_dec(flen, from, to, rsa, padding);
}
==============
But this code worked for 1.0.* branch

вт, 19 марта 2019 г., 19:52 Fuchs, Andreas <andreas.fuchs at sit.fraunhofer.de
>:

> Following scenario:
>
> I have an engine that implements e.g. RSA decryption.
> That engine links against a library that links against libcrypto to
> perform RSA decryption.
>
> Now if I have an application that sets the default library to be my
> engine, won't I end up in an infinite loop ?
>
> Or the other way around:
> - Is there a way to change the default engine from within my engine before
> calling out to the library ?
> - Can the library itself select "software-only" as an engine ?
>
> I guess I don't have a concrete problem in my specific case, since I have
> app_data attached to any key and
> if my engine receives a key without app_data, it will just call SSLeay()
> functions.
>
> But what if that was not the case ?
>
> Thanks a lot for any help,
> Andreas
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mta.openssl.org/pipermail/openssl-users/attachments/20190319/6ab49a3c/attachment-0001.html>