Engine with custom evp method callbacks

[Dmitry Belyavsky]

Dear Tobi,

On Fri, Nov 22, 2019 at 12:27 PM <Tobias.Wolf at t-systems.com> wrote:

> Hi everbody,
>
>
>
> I`m looking for a working example on how to implements a custom engine
> based on EVP methods callbacks. First I was implementing my custom engine
> based on RSA callbacks, but we found out that we cannot use this mechanism,
>
> therefore I need to change to EVP, details are written here
> https://github.com/openssl/openssl/issues/7968.
>
>
>
>                 RSA_METHOD* rsa_method = RSA_meth_new("OpenSSL Custom RSA
> method", 0);
>
>                 const RSA_METHOD* ossl_rsa_meth = RSA_PKCS1_OpenSSL();
>
>
>
>                 rc = RSA_meth_set_priv_enc(rsa_method,
> gk_openssl_rsa_priv_enc);
>
>
>
>                 rc = ENGINE_set_RSA(e, rsa_method);
>
>                 if (rc != TRUE) {
>
>                                return 0;
>
>                 }
>
>
>
>                 if (flags & ENGINE_METHOD_RSA) {
>
>                                rc = ENGINE_register_RSA(e);
>
>                                                if (rc != TRUE) {
>
>
> return 0;
>
>                                                }
>
>                 }
>
>
>
>
>
> Now I try with EVP the following source code but it’s not working:
>
>
>
> EVP_PKEY_METHOD* engine_pkey_methods = EVP_PKEY_meth_new(EVP_PKEY_RSA_PSS,
> 0);
>
> const EVP_PKEY_METHOD* ossl_pkey_methods = EVP_PKEY_meth_find(
> EVP_PKEY_RSA_PSS);
>
> EVP_PKEY_meth_copy(engine_pkey_methods, ossl_pkey_methods);
>
>
>
> // This shall be an equivalent to = RSA_PKCS1_OpenSSL();
>
> const EVP_PKEY_METHOD* ossl_pkey_methods = EVP_PKEY_meth_find(
> EVP_PKEY_RSA_PSS);
>
>
>
> But how to set the evp method the engine like RSA(e, rsa_method);?
>
> This expects another callback, but I just want to set the method?!
>
>
>
> int ENGINE_set_pkey_meths(ENGINE *e, ENGINE_PKEY_METHS_PTR f);
>
>
>

I strongly suppose that you can't mix EVP_PKEY_METHOD and RSA_METHOD, but
you should wrap the RSA_METHOD callbacks in the EVP_PKEY callbacks.
I suggest you look at the https://github.com/gost-engine/engine as an
example of providing the EVP operations via the engine.

I also have an example of providing custom RSA_METHOD somewhere but it was
designed to work with 1.0 and may be incompatible with the 1.1.* because of
using the internal structures.

-- 
SY, Dmitry Belyavsky
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mta.openssl.org/pipermail/openssl-users/attachments/20191123/4375dd23/attachment.html>