OpenSSL 1.1.1b - TLS server handshake fails when using CAPI engine - capi_rsa_priv_enc() - capi engine: function not supported

[sandeep m.v]

Hi,

This is regarding an issue reported here in this link -
https://github.com/openssl/openssl/issues/8872  - This is blocking my
development progress.
I'm seeing this issue - capi_rsa_priv_enc() - capi engine: function not
supported, when I tried to upgrade my application from using OpenSSL
version 1.0.2r to 1.1.1b.
This is causing TLS handshake to fail.
In my case, I'm creating a self signed certificate that is used by Server
application loading CAPI engine.
Certificate is created by calling CertCreateSelfSignCertificate(--) -
wincrypt.h - using szOID_RSA_SHA256RSA  signature algorithm and "Microsoft
Enhanced RSA and AES Cryptographic Provider".

This failure doesn't look like it's because of TLS1.3 as turning off TLS1.3
while configure with "no-tls1_3" also caused the same problem.
Here is the call stack that is causing the reported problem when
SSL_accept() is called.

 capi.dll!capi_rsa_priv_enc(int flen, const unsigned char * from, unsigned
char * to, rsa_st * rsa, int padding)
libcrypto-1_1.dll!RSA_private_encrypt(int flen, const unsigned char * from,
unsigned char * to, rsa_st * rsa, int padding)
libcrypto-1_1.dll!pkey_rsa_sign(evp_pkey_ctx_st * ctx, unsigned char * sig,
unsigned int * siglen, const unsigned char * tbs, unsigned int tbslen)
libcrypto-1_1.dll!EVP_PKEY_sign(evp_pkey_ctx_st * ctx, unsigned char * sig,
unsigned int * siglen, const unsigned char * tbs, unsigned int tbslen)
libcrypto-1_1.dll!EVP_DigestSignFinal(evp_md_ctx_st * ctx, unsigned char *
sigret, unsigned int * siglen)
libcrypto-1_1.dll!EVP_DigestSign(evp_md_ctx_st * ctx, unsigned char *
sigret, unsigned int * siglen, const unsigned char * tbs, unsigned int
tbslen)
libssl-1_1.dll!tls_construct_cert_verify(ssl_st * s, wpacket_st * pkt)
libssl-1_1.dll!write_state_machine(ssl_st * s)
libssl-1_1.dll!state_machine(ssl_st * s, int server)
libssl-1_1.dll!ossl_statem_accept(ssl_st * s)
libssl-1_1.dll!SSL_do_handshake(ssl_st * s)
libssl-1_1.dll!SSL_accept(ssl_st * s)



Is there a solution for this? Or
Do I need to switch to some other Signature algorithm like ECDSA? Can I use
this or anything else instead of RSA?
If I should switch to ECDSA, should I use "szOID_ECDSA_SHA256" (wincrypt.h)
as signature algorithm, use "PROV_EC_ECDSA_SIG" while calling
CryptAcquireContext() and call ENGINE_set_default() with ENGINE_METHOD_EC
to support ECDSA using capi.dll?

Thank you in advance.

-- 
Regards,
Sandeep
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mta.openssl.org/pipermail/openssl-users/attachments/20190715/7e2a9345/attachment.html>