[openssl-dev] confusion with rsa_meth_st in a custom RSA engine

[Brett R. Nicholas]

I just realized that my reply to Richard and Douglas didn't go through....I had the emails bundled to a digest, so not sure how to properly reply to an individual thread.  My prior email should have said "Thanks, I"ll give it a try and let you know how it goes". Disabled the digest now, though.


@Richard, I've looked at the source like you suggested. One more question......quoting your reply below:

> That flag means that the standard public/private encrypt/decrypt won't
> try to access the p, q, dmp1 and iqmp components of the RSA structure,
> i.e. the components that make up the private part.  Instead, that's
> left entirely to the mod_exp function (i.e. what you actually do implement).


This makes sense to me, and it seems that is the desired behavior. However, if I *only* reimplement the rsa_mod_exp() function, and leave the encrypt/decrypt functions to the default openSSL implementations, how can my engine know which of the four of those functions called its rsa_mod_exp function()? To put it another way: my accelerator will need to know whether it is meant to perform public key or private key operations, since the inputs will be written to different memory addresses. From within rsa_mod_exp(), the only way I can think of determining whether the function has been called by private_encrypt/decrypt() or public_encrypt/decrypt would be to check if the  p and q fields of the RSA *rsa struct passed into the function are NULL? But I can't tell from the source code if this will be guaranteed just by having  RSA_FLAG_EXT_PKEY set in the "flags" field of my engine's RSA_METHOD.


Thanks for the help so far!


Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mta.openssl.org/pipermail/openssl-dev/attachments/20170827/993edcef/attachment.html>