X509_sign_ctx and ENGINE

[Jan Just Keijser]

Hi,

On 13/04/21 17:05, Ron Kundla wrote:
> Hello,
>
> I am trying to adapt a piece of software to use an nCipher HSM using
> OpenSSL 1.1.1j along with the nfkm.dll engine library from nCipher.
>
> One function uses X509_sign_ctx() to calculate a SHA256 digest and
> sign a X509 certificate using RSA-PSS.
>
> I am running a debug version of x64 on Windows and I get the following
> error while executing the function rsa_pmeth.c:RSA_private_encrypt()
>
> error:06078081:digital envelope routines:EVP_PKEY_get0_DSA:expecting a dsa key
>
> This is the code that generates the error above:
>
>         if (rsa->d == NULL) {
>              RSAerr(RSA_F_RSA_OSSL_PRIVATE_ENCRYPT, RSA_R_MISSING_PRIVATE_KEY);
>              BN_free(d);
>              goto err;
>          }
>
> The debugger shows me that the RSA structure does not have any RSA key
> data inside of it, which would be due to the fact that it is in the
> HSM. I did use ENGINE_load_private_key() to create a PKEY that does
> show references to the engine structure and associated methods.
>
> My first guess is that X509_sign_ctx is not 'ENGINE' compatible. While
> the PKEY has an engine associated with it, there may not be the hooks
> to invoke the ENGINE private encryption method.
>
> Is there another way I can use OpenSSL functions to allow it to work
> with the ENGINE to do the exact same thing mentioned above? Or is the
> only way to make this work requrire X509_sign_ctx to be modified to
> support ENGINE operation?
>
>
I'd suggest to read the source code of the openssl 'req' command 
(./apps/req.c) , as it is capable of generating an X509 request from a 
token;  as a matter of fact, you will find that that code calls 
X509_sign_ct(...)

So I'd get the certificate request generation working first using
   openssl req -engine pkcs11 -keyform engine -key [<slotnr>:]12345678 
-new -text
    -out newcert.csr -config openssl.cnf -subj $SUBJECT

first with the nCipher HSM and if that works, then look at your code.

HTH,

JJK