Engine with custom evp method callbacks

[Tobias.Wolf at t-systems.com]

Hi everbody,

I`m looking for a working example on how to implements a custom engine based on EVP methods callbacks. First I was implementing my custom engine based on RSA callbacks, but we found out that we cannot use this mechanism,
therefore I need to change to EVP, details are written here https://github.com/openssl/openssl/issues/7968.

                RSA_METHOD* rsa_method = RSA_meth_new("OpenSSL Custom RSA method", 0);
                const RSA_METHOD* ossl_rsa_meth = RSA_PKCS1_OpenSSL();

                rc = RSA_meth_set_priv_enc(rsa_method, gk_openssl_rsa_priv_enc);

                rc = ENGINE_set_RSA(e, rsa_method);
                if (rc != TRUE) {
                               return 0;
                }

                if (flags & ENGINE_METHOD_RSA) {
                               rc = ENGINE_register_RSA(e);
                                               if (rc != TRUE) {
                                                                               return 0;
                                               }
                }


Now I try with EVP the following source code but it's not working:

EVP_PKEY_METHOD* engine_pkey_methods = EVP_PKEY_meth_new(EVP_PKEY_RSA_PSS, 0);
const EVP_PKEY_METHOD* ossl_pkey_methods = EVP_PKEY_meth_find(EVP_PKEY_RSA_PSS);
EVP_PKEY_meth_copy(engine_pkey_methods, ossl_pkey_methods);

// This shall be an equivalent to = RSA_PKCS1_OpenSSL();
const EVP_PKEY_METHOD* ossl_pkey_methods = EVP_PKEY_meth_find(EVP_PKEY_RSA_PSS);

But how to set the evp method the engine like RSA(e, rsa_method);?
This expects another callback, but I just want to set the method?!

int ENGINE_set_pkey_meths(ENGINE *e, ENGINE_PKEY_METHS_PTR f);

regards
Tobi



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mta.openssl.org/pipermail/openssl-users/attachments/20191122/e240fdae/attachment-0001.html>