Handshake Failure : SSL_accept:Error in before SSL initialization

[Matt Caswell]

Does errno give you anything?

How did you create your BIOs for m_pSsl?

Matt

On 04/02/2022 16:25, Kamala Ayyar wrote:
> Hello Matt,
> 
> The SSL_get_error() returns 5(SSL_ERROR_SYSCALL) It does not print 
> anything for this error, just an empty string.
> I use the following to print error but nothing is printed
> if ((retVal = SSL_accept(m_pSsl)) < 1)
> {
> sslError = SSL_get_error(m_pSsl, retVal);
> LOGERROR(getOpenSSLError());
> throw dwRet;// eSSL_ERROR;
> }
> string getOpenSSLError()
> {
> BIO *bio = BIO_new(BIO_s_mem());
> ERR_print_errors(bio);
> char *buf;
> size_t len = BIO_get_mem_data(bio, &buf);
> string ret(buf, len);
> BIO_free(bio);
> return ret;
> }
> 
> *Kamala  Ayyar*
> 502 Claremont Ave.
> Teaneck NJ 07666-2563
> Tel: (201)530-0861
> 
> 
> On Fri, Feb 4, 2022 at 10:54 AM Matt Caswell <matt at openssl.org 
> <mailto:matt at openssl.org>> wrote:
> 
> 
> 
>     On 04/02/2022 15:17, Kamala Ayyar wrote:
>      >
>      > Hello,
>      >
>      > We are facing a strange handshake failure issue with a test
>     server and
>      > client application using OpenSSL in Windows.  We have tried with
>     both
>      > 1.1.1g and 3.0.1 versions- same problem. We created a Dll to
>     handle the
>      > OpenSSL functions- where the SSL context, SSL object and
>     certificates
>      > are handled. The certificates are obtained from the Windows store
>     and
>      > converted to cert and key using PKCS12_parse()
>      > The server accepts non secure connection from the client and then
>     passes
>      > the socket to the Dll that calls the TLS_server_method() and
>     creates the
>      > SSL context, SSL object and loads the certificates for use. It
>     however
>      > fails at SSL_accept(m_pSsl). We use a call
>      > back SSL_set_info_callback(m_pSsl, apps_ssl_info_callback) that
>     gave us
>      > the following error information
>      > SSL_accept:Error in before SSL initialization
>      > On the client side the same Dll is called with a client
>      > method TLS_client_method() and the error displayed
>     is SSL_connect:Error
>      > in SSLv3/TLS write client hello
>      > We have confirmed the certificates are good and valid.
>      >
>      > The same Dll called from a different heavily threaded application
>     with
>      > over 2000+ clients works well and handshake connections established
>      > without issues on a different port number.
>      >
>      > We have also tried to use OpenSSL methods directly without using
>     the Dll
>      > but we get the same failure.  This was also used with server and
>     client
>      > on the same machine as well as different machines with the same
>      > outcome.  The non secure communication works fine between the
>     server and
>      > the client
> 
>     What does SSL_get_error() report after SSL_accept() fails?
> 
>     Also please dump the OpenSSL error stack when it fails, e.g. using
>     something like ERR_print_errors_fp(stdout);
> 
>     Matt
>