X509_sign_ctx and ENGINE

[Ron Kundla]

Hello,

I am trying to adapt a piece of software to use an nCipher HSM using
OpenSSL 1.1.1j along with the nfkm.dll engine library from nCipher.

One function uses X509_sign_ctx() to calculate a SHA256 digest and
sign a X509 certificate using RSA-PSS.

I am running a debug version of x64 on Windows and I get the following
error while executing the function rsa_pmeth.c:RSA_private_encrypt()

error:06078081:digital envelope routines:EVP_PKEY_get0_DSA:expecting a dsa key

This is the code that generates the error above:

       if (rsa->d == NULL) {
            RSAerr(RSA_F_RSA_OSSL_PRIVATE_ENCRYPT, RSA_R_MISSING_PRIVATE_KEY);
            BN_free(d);
            goto err;
        }

The debugger shows me that the RSA structure does not have any RSA key
data inside of it, which would be due to the fact that it is in the
HSM. I did use ENGINE_load_private_key() to create a PKEY that does
show references to the engine structure and associated methods.

My first guess is that X509_sign_ctx is not 'ENGINE' compatible. While
the PKEY has an engine associated with it, there may not be the hooks
to invoke the ENGINE private encryption method.

Is there another way I can use OpenSSL functions to allow it to work
with the ENGINE to do the exact same thing mentioned above? Or is the
only way to make this work requrire X509_sign_ctx to be modified to
support ENGINE operation?

Thanks,
Ron