Handshake Failure : SSL_accept:Error in before SSL initialization

[Matt Caswell]

On 04/02/2022 15:17, Kamala Ayyar wrote:
> 
> Hello,
> 
> We are facing a strange handshake failure issue with a test server and 
> client application using OpenSSL in Windows.  We have tried with both 
> 1.1.1g and 3.0.1 versions- same problem. We created a Dll to handle the 
> OpenSSL functions- where the SSL context, SSL object and certificates 
> are handled. The certificates are obtained from the Windows store and 
> converted to cert and key using PKCS12_parse()
> The server accepts non secure connection from the client and then passes 
> the socket to the Dll that calls the TLS_server_method() and creates the 
> SSL context, SSL object and loads the certificates for use. It however 
> fails at SSL_accept(m_pSsl). We use a call 
> back SSL_set_info_callback(m_pSsl, apps_ssl_info_callback) that gave us 
> the following error information
> SSL_accept:Error in before SSL initialization
> On the client side the same Dll is called with a client 
> method TLS_client_method() and the error displayed is SSL_connect:Error 
> in SSLv3/TLS write client hello
> We have confirmed the certificates are good and valid.
> 
> The same Dll called from a different heavily threaded application with 
> over 2000+ clients works well and handshake connections established 
> without issues on a different port number.
> 
> We have also tried to use OpenSSL methods directly without using the Dll 
> but we get the same failure.  This was also used with server and client 
> on the same machine as well as different machines with the same 
> outcome.  The non secure communication works fine between the server and 
> the client

What does SSL_get_error() report after SSL_accept() fails?

Also please dump the OpenSSL error stack when it fails, e.g. using 
something like ERR_print_errors_fp(stdout);

Matt