OpenSSL 1.1.1b - TLS server handshake fails when using CAPI engine - capi_rsa_priv_enc() - capi engine: function not supported

[Michael Wojcik]

> From: openssl-users [mailto:openssl-users-bounces at openssl.org] On Behalf Of sandeep m.v
> Sent: Monday, July 15, 2019 11:56

> I'm seeing this issue - capi_rsa_priv_enc() - capi engine: function not supported,
> when I tried to upgrade my application from using OpenSSL version 1.0.2r to 1.1.1b.
> This is causing TLS handshake to fail.
> In my case, I'm creating a self signed certificate that is used by Server application
> loading CAPI engine.
> Certificate is created by calling CertCreateSelfSignCertificate(--) - wincrypt.h -
> using szOID_RSA_SHA256RSA  signature algorithm and "Microsoft Enhanced RSA and
> AES Cryptographic Provider".

It's been some years since I worked with OpenSSL CAPI support, and in particular I haven't done anything with the CAPI engine for 1.1.

For 1.0.2, though, I ended up forking the OpenSSL CAPI engine support and enhancing it in a number of places. I changed capi_load_privkey and the functions it calls (capi_find_key and capi_get_pkey) to silently determine if the provider type in the context was wrong, and if so correct it.

CAPI is a fairly horrible API (CNG is somewhat better), and in particular is very fragile when there's a mismatch between the provider type in the CAPI context and the provider type specified for the key. It may be that your CAPI context specifies a provider other than the Enhanced RSA and AES one.

The CAPI engine for 1.0.2 (at least the version I forked from) also had a shortcoming which Steve Henson had suggested a fix for, but which wasn't in the code. That's down in capi_get_key where it calls CryptGetUserKey. If CryptGetUserKey fails with NTE_NO_KEY (you have to call GetLastError to check), then try again with the other key type, by XORing keyspec with 3:

   if (!CryptGetUserKey(key->hprov, keyspec, &key->key))
      {
      success = FALSE;
      errcode = GetLastError();
      if (errcode == NTE_NO_KEY)
        {
        keyspec ^= 3;
        success = CryptGetUserKey(key->hprov, keyspec, &key->key);
        }
      ...
      }

This handles the case where you have an RSA key marked as a key-exchange key and you need a signing key, or vice versa.

All of that may be fixed already in the 1.1 CAPI engine, or be irrelevant to your problem, of course. But those are potential issues when using RSA keys with CAPI.

--
Michael Wojcik
Distinguished Engineer, Micro Focus