X509_sign_ctx and ENGINE

[Ron Kundla]

Thanks Jan, I'll check it out and let you know how it goes.

Regards,
Ron

On Tue, Apr 13, 2021 at 12:18 PM Jan Just Keijser <janjust at nikhef.nl> wrote:
>
> Hi,
>
> On 13/04/21 17:05, Ron Kundla wrote:
> > Hello,
> >
> > I am trying to adapt a piece of software to use an nCipher HSM using
> > OpenSSL 1.1.1j along with the nfkm.dll engine library from nCipher.
> >
> > One function uses X509_sign_ctx() to calculate a SHA256 digest and
> > sign a X509 certificate using RSA-PSS.
> >
> > I am running a debug version of x64 on Windows and I get the following
> > error while executing the function rsa_pmeth.c:RSA_private_encrypt()
> >
> > error:06078081:digital envelope routines:EVP_PKEY_get0_DSA:expecting a dsa key
> >
> > This is the code that generates the error above:
> >
> >         if (rsa->d == NULL) {
> >              RSAerr(RSA_F_RSA_OSSL_PRIVATE_ENCRYPT, RSA_R_MISSING_PRIVATE_KEY);
> >              BN_free(d);
> >              goto err;
> >          }
> >
> > The debugger shows me that the RSA structure does not have any RSA key
> > data inside of it, which would be due to the fact that it is in the
> > HSM. I did use ENGINE_load_private_key() to create a PKEY that does
> > show references to the engine structure and associated methods.
> >
> > My first guess is that X509_sign_ctx is not 'ENGINE' compatible. While
> > the PKEY has an engine associated with it, there may not be the hooks
> > to invoke the ENGINE private encryption method.
> >
> > Is there another way I can use OpenSSL functions to allow it to work
> > with the ENGINE to do the exact same thing mentioned above? Or is the
> > only way to make this work requrire X509_sign_ctx to be modified to
> > support ENGINE operation?
> >
> >
> I'd suggest to read the source code of the openssl 'req' command
> (./apps/req.c) , as it is capable of generating an X509 request from a
> token;  as a matter of fact, you will find that that code calls
> X509_sign_ct(...)
>
> So I'd get the certificate request generation working first using
>    openssl req -engine pkcs11 -keyform engine -key [<slotnr>:]12345678
> -new -text
>     -out newcert.csr -config openssl.cnf -subj $SUBJECT
>
> first with the nCipher HSM and if that works, then look at your code.
>
> HTH,
>
> JJK
>