[openssl-users] CAPI-Engine doc

[Richard Oehlinger]

On 10/23/2018 05:22 PM, Selva Nair wrote:
> On Tue, Oct 23, 2018 at 10:38 AM Richard Oehlinger via openssl-users
> <openssl-users at openssl.org> wrote:
>> Hi!
>>
>> I'm trying to get a handle on the CAPI engine, because I need to have a
>> secure Keystore on Windows. Furthermore I need it to work with Qt's
>> QSslKey, which fortunately can be constructed by EVP_PKEY *.
>>
>> So far so good. The key is found, but when I try to use it in a SSL
>> connection i get following error:
>>
>> error:80070063:lib(128):CAPI_RSA_SIGN:cant create hash object,
>> error:1409B006:SSL routines:ssl3_send_server_key_exchange:EVP lib
> Which version of OpenSSL?
I use 1.0.2p.
>
>> Trace Output is:
>>
>> Setting debug file to C:\Users\user\AppData\Local\Temp\engine.txt
>> Opening certificate store MY
>> capi_get_key, contname={4EBA52A8-AB4B-47DB-B777-2B26351F324C},
>> provname=Microsoft Enhanced Cryptographic Provider v1.0, type=1
>> Called CAPI_rsa_sign()
> This CSP cannot do SHA2 hashes so won't work unless you restrict
> signature algorithms or set TLS version to 1.1. I believe OpenSSL
> 1.1.0 will try to load The ".. Enhanced RSA AES .. Provider" which
> can handle SHA2 and may work. I say "may" because, if the key store is
> a legacy hardware token, it also depends on signature algorithms supported
> by the token and may be necessary to downgrade to TLS 1.1.
>
> Selva
Yes this did the trick, when forced the TLS version 1.1 the key did work.
Unfortunately I've requirement of 1.2 on my project.
I managed to set the provider name from outside when using a different 
lookup method:

     assert(ENGINE_ctrl(engine, CAPI_CMD_SET_CSP_TYPE, PROV_RSA_AES, 0, 0));
     assert(ENGINE_ctrl(engine, CAPI_CMD_SET_CSP_NAME, 0, 
(void*)MS_ENH_RSA_AES_PROV, 0))
     assert(ENGINE_ctrl(engine, CAPI_CMD_LOOKUP_METHOD, 
CAPI_LU_CONTNAME, 0,0));
     EVP_PKEY *key = ENGINE_load_private_key(engine, 
"{4EBA52A8-AB4B-47DB-B777-2B26351F324C}", NULL, NULL);

Now I need to somehow lookup the key name myself, but at least it works 
with TLS 1.2 now.

Thank you for all your help!
Regards,
Richard