[openssl-users] How to override methods in EVP_PKEY_METHOD structure that is attached to a EVP_PKEY_CTX?

[Dr. Stephen Henson]

On Fri, Feb 17, 2017, Stephan M?hlstrasser wrote:

> Hi,
> 
> we use OpenSSL 1.0.2 together with PKCS#11 tokens by plugging
> methods into the RSA_METHOD structure that interface with the
> PKCS#11 token, and this works fine so far. However, for creating RSA
> signatures with PSS padding this strategy doesn't work anymore,
> because OpenSSL wants to directly encrypt with the private key in
> this case, which is not possible in PKCS#11.
> 
> Therefore my idea is to override the function pkey_rsa_sign() and
> plug a wrapper around it into the EVP_PKEY_METHOD structure that is
> associated with the EVP_PKEY_CTX to handle this special situation.
> 
> The header evp.h declares the following functions among others:
> 
> EVP_PKEY_METHOD* EVP_PKEY_meth_new(int id, int flags);
> void EVP_PKEY_meth_copy(EVP_PKEY_METHOD *dst, const EVP_PKEY_METHOD *src);
> 
> void EVP_PKEY_meth_set_sign(EVP_PKEY_METHOD *pmeth,
> 	int (*sign_init)(EVP_PKEY_CTX *ctx),
> 	int (*sign)(EVP_PKEY_CTX *ctx, unsigned char *sig,
>             size_t *siglen, const unsigned char *tbs, size_t tbslen));
> 
> But I can't figure out how to use these functions to achieve what I
> want, because the following pieces seem to be missing:
> 
> - Retrieve the EVP_PKEY_METHOD pointer from a EVP_PKEY_CTX pointer
> - Set the EVP_PKEY_METHOD pointer for a EVP_PKEY_CTX pointer
> - Retrieve the existing "sign_init" and "sign" function pointers
> from an initialized EVP_PKEY_METHOD pointer for being able to wrap
> them
> 
> Is it possible to override methods in an EVP_PKEY_METHOD structure,
> or would it be necessary to implement a whole OpenSSL engine to do
> what I want?
> 

It should be possible yes, though AFAIK no one has yet tried to do this so
there may be some pieces missing.

In outline you'd retrieve the appropriate EVP_PKEY_METHOD for the algorithm of
interest, make a copy of it and then set the operation you wish to override,
you can also retrieve the original operation in case you sometimes wish to
call that.

Steve.
--
Dr Stephen N. Henson. OpenSSL project core developer.
Commercial tech support now available see: http://www.openssl.org