[openssl-users] confusion with rsa_meth_st in a custom RSA engine

[Brett R. Nicholas]

I am trying to develop a engine for a custom RSA hardware accelerator, and have a few questions about the RSA_METHOD stucture implementation.


Some context: for encryption, my accelerator takes as inputs the base, public exponent, and modulus, and returns the resulting ciphertext. For decryption, it takes as inputs the base, and modulus. It does not need a private key, as this is stored in hardware, and can only be configured through an out-of-band channel. I already have a kernel module that exposes an API to userspace programs to use the accelerator. Now I just need to integrate it into openSSL.


I've already created a similar engine for AES and SHA256, however I'm struggling with RSA. Ideally, I'd like to not have to worry about anything other than just performing the modular exponentiation on a pre-padded and prepared chunk of data. For SHA and AES, this is straightforward: all that was taken care of by the EVP interface, so all I needed to worry about was getting the data two and from my accelerators. But it doesn't appear to be as simple for RSA (pls correct me if I'm wrong).


I'm confused as to which RSA_METHOD function pointers that my engine needs to implement.  I show the structure below for reference:

struct rsa_meth_st {
    char *name;
    int (*rsa_pub_enc) (int flen, const unsigned char *from,
                        unsigned char *to, RSA *rsa, int padding);
    int (*rsa_pub_dec) (int flen, const unsigned char *from,
                        unsigned char *to, RSA *rsa, int padding);
    int (*rsa_priv_enc) (int flen, const unsigned char *from,
                         unsigned char *to, RSA *rsa, int padding);
    int (*rsa_priv_dec) (int flen, const unsigned char *from,
                         unsigned char *to, RSA *rsa, int padding);

    int (*rsa_mod_exp) (BIGNUM *r0, const BIGNUM *I, RSA *rsa, BN_CTX *ctx);

    int (*bn_mod_exp) (BIGNUM *r, const BIGNUM *a, const BIGNUM *p,
                       const BIGNUM *m, BN_CTX *ctx, BN_MONT_CTX *m_ctx);
    /* ....stuff.... */
    int flags;
    /* .... stuff ... */
};  // TYPEDEF'ED TO RSA_METHOD in include/ossl_typ.h


So, three questions:


  1.  Is it possible for the standard OpenSSL RSA implementation to use my engine's "modular exponentiation" function, without having to rewrite the RSA_[public|private]_[encrypt|decrypt] family of functions from /include/openssl/rsa.h?
  2.  If so, does it suffice to only implement the rsa_mod_exp function? Or must I implement both public_enc/dec and private_enc/dec functions as well? I ask, because the source code for the old Intel RSAX engine (https://gist.github.com/bigbrett/91903f773f9d150b7329c7d462cd220a) does this, but I can't figure out how and when in the "RSA flow" the engine's function gets invoked.
  3.  In  /include/openssl/rsa.h, I saw the following macro for the RSA_METHOD flag field (line 55):

/*
 * This flag means the private key operations will be handled by rsa_mod_exp
 * and that they do not depend on the private key components being present:
 * for example a key stored in external hardware. Without this flag
 * bn_mod_exp gets called when private key components are absent.
 */
# define RSA_FLAG_EXT_PKEY               0x0020


Does this mean that if I use this flag in the "flags" field of RSA_METHOD, that I DO NOT need to implement rsa_pub_enc/dec and friends? I guess I'm just confused as to at what point in the RSA encryption/decryption process my engine should be invoked at.


FWIW I'm planning on releasing a comprehensive engine tutorial and some documentation for all this stuff once I finish (will already be written up in my thesis), so I want to make sure I get these details ironed out!


Any words of wisdom would be greatly appreciated.


Best,

Brett

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mta.openssl.org/pipermail/openssl-users/attachments/20170823/17064b4b/attachment.html>