[openssl-dev] confusion with rsa_meth_st in a custom RSA engine

[Douglas E Engert]

You can use  RSA_meth_dup to copy the default RSA_METHOD,then use the RSA_meth_set* routines to point to your own routines.
it only overrides  RSA_meth_set1_name, RSA_meth_set_priv_enc RSA_meth_set_priv_dec RSA_meth_set_finish see:

https://github.com/OpenSC/libp11/blob/master/src/p11_rsa.c

But You will have to populate the key with the public parts of the key.



On 8/22/2017 8:50 PM, Brett R. Nicholas wrote:
> I am trying to develop a engine for a custom RSA hardware accelerator, and have a few questions about the RSA_METHOD stucture implementation.
> 
> 
> Some context: for encryption, my accelerator takes as inputs the base, public exponent, and modulus, and returns the resulting ciphertext. For decryption, it takes as inputs the base, and modulus. It 
> does not need a private key, as this is stored in hardware, and can only be configured through an out-of-band channel. I already have a kernel module that exposes an API to userspace programs to use 
> the accelerator. Now I just need to integrate it into openSSL.
> 
> 
> I've already created a similar engine for AES and SHA256, however I'm struggling with RSA. Ideally, I'd like to not have to worry about anything other than just performing the modular exponentiation 
> on a pre-padded and prepared chunk of data. For SHA and AES, this is straightforward: all that was taken care of by the EVP interface, so all I needed to worry about was getting the data two and from 
> my accelerators. But it doesn't appear to be as simple for RSA (pls correct me if I'm wrong).
> 
> 
> I'm confused as to which RSA_METHOD function pointers that my engine needs to implement.  I show the structure below for reference:
> 
> struct rsa_meth_st {
>      char *name;
>      int (*rsa_pub_enc) (int flen, const unsigned char *from,
>                          unsigned char *to, RSA *rsa, int padding);
>      int (*rsa_pub_dec) (int flen, const unsigned char *from,
>                          unsigned char *to, RSA *rsa, int padding);
>      int (*rsa_priv_enc) (int flen, const unsigned char *from,
>                           unsigned char *to, RSA *rsa, int padding);
>      int (*rsa_priv_dec) (int flen, const unsigned char *from,
>                           unsigned char *to, RSA *rsa, int padding);
> 
>      int (*rsa_mod_exp) (BIGNUM *r0, const BIGNUM *I, RSA *rsa, BN_CTX *ctx);
> 
>      int (*bn_mod_exp) (BIGNUM *r, const BIGNUM *a, const BIGNUM *p,
>                         const BIGNUM *m, BN_CTX *ctx, BN_MONT_CTX *m_ctx);
>      /* ....stuff.... */
>      int flags;
>      /* .... stuff ... */
> };  // TYPEDEF'ED TO RSA_METHOD in include/ossl_typ.h
> 
> 
> *So, three questions:
> *
> 
> *
> *
> 
>  1. *Is it possible for the standard OpenSSL RSA implementation to use my engine's "modular exponentiation" function, without having to rewrite the RSA_[public|private]_[encrypt|decrypt] family of
>     functions from /include/openssl/rsa.h?
>     *
>  2. *If so, does it suffice to only implement the rsa_mod_exp function? *Or must I implement both public_enc/dec and private_enc/dec functions as well? I ask, because the source code for the old Intel
>     RSAX engine (https://gist.github.com/bigbrett/91903f773f9d150b7329c7d462cd220a) does this, but I can't figure out how and when in the "RSA flow" the engine's function gets invoked.
>  3. In /include/openssl/rsa.h, I saw the following macro for the RSA_METHOD flag field (line 55):
> 
> /*
>   * This flag means the private key operations will be handled by rsa_mod_exp
>   * and that they do not depend on the private key components being present:
>   * for example a key stored in external hardware. Without this flag
>   * bn_mod_exp gets called when private key components are absent.
>   */
> # define RSA_FLAG_EXT_PKEY               0x0020
> 
> *Does this mean that if I use this flag in the "flags" field of RSA_METHOD, that I DO NOT need to implement rsa_pub_enc/dec and friends? *I guess I'm just confused as to at what point in the RSA 
> encryption/decryption process my engine should be invoked at.
> 
> 
> FWIW I'm planning on releasing a comprehensive engine tutorial and some documentation for all this stuff once I finish (will already be written up in my thesis), so I want to make sure I get these 
> details ironed out!
> 
> 
> Any words of wisdom would be greatly appreciated.
> 
> Best,
> 
> Brett
> 
> 
> 
> 

-- 

  Douglas E. Engert  <DEEngert at gmail.com>