[openssl-dev] confusion with rsa_meth_st in a custom RSA engine

[Richard Levitte]

In message <BN3PR03MB1462808DC39E3A2A1386AEDBE0850 at BN3PR03MB1462.namprd03.prod.outlook.com> on Wed, 23 Aug 2017 01:50:04 +0000, "Brett R. Nicholas" <Brett.R.Nicholas.TH at dartmouth.edu> said:

> I am trying to develop a engine for a custom RSA hardware accelerator, and have a few questions
> about the RSA_METHOD stucture implementation.
...
> I'm confused as to which RSA_METHOD function pointers that my engine needs to implement. I
> show the structure below for reference:
> 
> struct rsa_meth_st {
> char *name;
> int (*rsa_pub_enc) (int flen, const unsigned char *from,
> unsigned char *to, RSA *rsa, int padding);
> int (*rsa_pub_dec) (int flen, const unsigned char *from,
> unsigned char *to, RSA *rsa, int padding);
> int (*rsa_priv_enc) (int flen, const unsigned char *from,
> unsigned char *to, RSA *rsa, int padding);
> int (*rsa_priv_dec) (int flen, const unsigned char *from,
> unsigned char *to, RSA *rsa, int padding);
> 
> int (*rsa_mod_exp) (BIGNUM *r0, const BIGNUM *I, RSA *rsa, BN_CTX *ctx);
> 
> int (*bn_mod_exp) (BIGNUM *r, const BIGNUM *a, const BIGNUM *p,
> const BIGNUM *m, BN_CTX *ctx, BN_MONT_CTX *m_ctx);
> /* ....stuff.... */
> int flags;
> /* .... stuff ... */
> }; // TYPEDEF'ED TO RSA_METHOD in include/ossl_typ.h
> 
> So, three questions:
> 
> 1 Is it possible for the standard OpenSSL RSA implementation to use my engine's
>  "modular exponentiation" function, without having to rewrite the RSA_
>  [public|private]_[encrypt|decrypt] family of functions from /include/openssl/rsa.h?

Yes.

> 2 If so, does it suffice to only implement the rsa_mod_exp function? Or must I implement
>  both public_enc/dec and private_enc/dec functions as well? I ask, because the source code for
>  the old Intel RSAX engine
>  (https://gist.github.com/bigbrett/91903f773f9d150b7329c7d462cd220a) does this, but I can't
>  figure out how and when in the "RSA flow" the engine's function gets invoked.

I'd like to point out this part of the code, which is relevant:

    #ifndef OPENSSL_NO_RSA
    	meth1 = RSA_PKCS1_SSLeay();
    	e_rsax_rsa.rsa_pub_enc = meth1->rsa_pub_enc;
    	e_rsax_rsa.rsa_pub_dec = meth1->rsa_pub_dec;
    	e_rsax_rsa.rsa_priv_enc = meth1->rsa_priv_enc;
    	e_rsax_rsa.rsa_priv_dec = meth1->rsa_priv_dec;
    	e_rsax_rsa.bn_mod_exp = meth1->bn_mod_exp;
    	e_rsax_rsa.finish = meth1->finish;
    #endif

You you see, you don't need to actually *reimplement* the
public/private encrypt/decrypt functions, but you need to make sure
there's *some* implementation available through the method you build
up.  Incidently, in more modern OpenSSLs, you should use
RSA_PKCS1_OpenSSL() rather than RSA_PKCS1_SSLeay().

Anyhow, the standard public/private encrypt/decrypt functions will
call the mod_exp functions that you implement

> 3 In /include/openssl/rsa.h, I saw the following macro for the RSA_METHOD flag field (line 55):
> 
> /*
> * This flag means the private key operations will be handled by rsa_mod_exp
> * and that they do not depend on the private key components being present:
> * for example a key stored in external hardware. Without this flag
> * bn_mod_exp gets called when private key components are absent.
> */
> # define RSA_FLAG_EXT_PKEY 0x0020
> 
> Does this mean that if I use this flag in the "flags" field of RSA_METHOD, that I DO NOT
> need to implement rsa_pub_enc/dec and friends? I guess I'm just confused as to at what
> point in the RSA encryption/decryption process my engine should be invoked at.

That flag means that the standard public/private encrypt/decrypt won't
try to access the p, q, dmp1 and iqmp components of the RSA structure,
i.e. the components that make up the private part.  Instead, that's
left entirely to the mod_exp function (i.e. what you actually do
implement).

If you want to see for yourself what's happening, I suggest a study of
crypto/rsa/rsa_ossl.c (OpenSSL 1.1.0 and up) or crypro/rsa/rsa_eay.c
(OpenSSL 1.0.2).

Cheers,
Richard

-- 
Richard Levitte         levitte at openssl.org
OpenSSL Project         http://www.openssl.org/~levitte/