[openssl-users] CAPI-Engine doc

[Michael Wojcik]

> From: openssl-users [mailto:openssl-users-bounces at openssl.org] On Behalf
> Of Richard Oehlinger via openssl-users
> Sent: Tuesday, October 23, 2018 10:38
>
> I'm trying to get a handle on the CAPI engine, because I need to have a
> secure Keystore on Windows. Furthermore I need it to work with Qt's
> QSslKey, which fortunately can be constructed by EVP_PKEY *.

What OpenSSL version are you using? Please remember to include this informtion in every question. (And, normally, we'd ask for the platform as well, but since CAPI is Windows-specific, we know that in this case.)

> So far so good. The key is found, but when I try to use it in a SSL
> connection i get following error:
>
> error:80070063:lib(128):CAPI_RSA_SIGN:cant create hash object,
> error:1409B006:SSL routines:ssl3_send_server_key_exchange:EVP lib
>
> I use a current Windows 10. Do I need to use a different Algorithm in
> order to work? Some googeling is indicating the provider might be wrong.

I haven't looked at the CAPI engine code since 1.0.1j. At that time, I needed CAPI support and discovered there were various issues with the extant CAPI code, so I forked and rewrote it. That was some time back, obviously, and I'm afraid I never got around to pushing the changes back to openssl.org. (In fact, it was sufficiently long ago that I believe the organization was still reluctant to take contributions from people in the US at the time.)

The biggest issue was with provider handling. CAPI is something of a braindead API in many ways - Microsoft's replacement, CNG, is somewhat better - and the provider stuff is one of them. When a key (including a "key" which is actually just a reference to a key contained in an HSM) is imported into one of the Windows key stores, it has to be associated with a provider, and that provider has to accommodate that type and size of key; otherwise the key is unusable. Then, when you try to use the key in CAPI, you have to specify the same provider - CAPI isn't smart enough to figure it out on its own.

So my version of the CAPI engine has code to look up the key's provider and silently correct the provider type information in the engine's context structure if it's a mismatch.

Beyond that, it appears that my changes included:

- Support for building all the necessary functionality when using Microsoft Windows SDK 6.0A, which was one of my requirements at the time.

- Supporting hashes other than SHA-1 for DSA. We have US Federal customers who needed fairly comprehensive DSA support. For most people this is likely a non-issue.

- Forcing stack probes on for the callback functions, because my engine was being built outside the OpenSSL build process, but needed to match the calling convention of OpenSSL, which (at least in 1.0.1j) included stack-probe support.

- A fix suggested by Steven Henson years ago on the mailing list to capi_get_key, but never (at least by 1.0.1j) picked up in the source code: If CryptGetUserKey returns NTE_NO_KEY, xor keyspec with 3 to flip the key type and try CryptGetUesrKey again.

I think that's it, though it's possible I tweaked some other things and didn't call them out in the comments.

I suppose I should check what the CAPI engine source looks like in 1.1.1, merge my changes in if feasible, and submit a PR. One of these days...

Really, though, what we need is a new engine written to use CNG rather than CAPI. Though that would have the disadvantage of not supporting ancient Windows OS and SDK versions which, while unsupported by Microsoft, are still used in far too many places.

--
Michael Wojcik
Distinguished Engineer, Micro Focus