[openssl-dev] confusion with rsa_meth_st in a custom RSA engine

[Dr. Stephen Henson]

On Sun, Aug 27, 2017, Brett R. Nicholas wrote:

> 
> This makes sense to me, and it seems that is the desired behavior. However,
> if I *only* reimplement the rsa_mod_exp() function, and leave the
> encrypt/decrypt functions to the default openSSL implementations, how can my
> engine know which of the four of those functions called its rsa_mod_exp
> function()? To put it another way: my accelerator will need to know whether
> it is meant to perform public key or private key operations, since the
> inputs will be written to different memory addresses. From within
> rsa_mod_exp(), the only way I can think of determining whether the function
> has been called by private_encrypt/decrypt() or public_encrypt/decrypt would
> be to check if the  p and q fields of the RSA *rsa struct passed into the
> function are NULL? But I can't tell from the source code if this will be
> guaranteed just by having  RSA_FLAG_EXT_PKEY set in the "flags" field of my
> engine's RSA_METHOD.
> 

The rsa_mod_exp function is only called for private key operations. You can't
tell if it is a private encrypt or a private decrypt though but that
shouldn't matter because the operation performed at that level is the same for
both.

Steve.
--
Dr Stephen N. Henson. OpenSSL project core developer.
Commercial tech support now available see: http://www.openssl.org