Handshake Failure : SSL_accept:Error in before SSL initialization

[Kamala Ayyar]

Hello,

We are facing a strange handshake failure issue with a test server and
client application using OpenSSL in Windows.  We have tried with both
1.1.1g and 3.0.1 versions- same problem. We created a Dll to handle the
OpenSSL functions- where the SSL context, SSL object and certificates are
handled. The certificates are obtained from the Windows store and converted
to cert and key using PKCS12_parse()
The server accepts non secure connection from the client and then passes
the socket to the Dll that calls the TLS_server_method() and creates the
SSL context, SSL object and loads the certificates for use. It however
fails at SSL_accept(m_pSsl). We use a call
back SSL_set_info_callback(m_pSsl, apps_ssl_info_callback) that gave us the
following error information
SSL_accept:Error in before SSL initialization
On the client side the same Dll is called with a client
method TLS_client_method() and the error displayed is SSL_connect:Error in
SSLv3/TLS write client hello
We have confirmed the certificates are good and valid.

The same Dll called from a different heavily threaded application with over
2000+ clients works well and handshake connections established
without issues on a different port number.

We have also tried to use OpenSSL methods directly without using the Dll
but we get the same failure.  This was also used with server and client on
the same machine as well as different machines with the same outcome.  The
non secure communication works fine between the server and the client

Thanks
Kamala
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mta.openssl.org/pipermail/openssl-users/attachments/20220204/9b6ed7da/attachment.htm>