Handshake Failure : SSL_accept:Error in before SSL initialization

[Kamala Ayyar]

Hello Matt,

The SSL_get_error() returns 5(SSL_ERROR_SYSCALL) It does not print anything
for this error, just an empty string.
I use the following to print error but nothing is printed
if ((retVal = SSL_accept(m_pSsl)) < 1)
{
sslError = SSL_get_error(m_pSsl, retVal);
LOGERROR(getOpenSSLError());
throw dwRet;// eSSL_ERROR;
}
string getOpenSSLError()
{
BIO *bio = BIO_new(BIO_s_mem());
ERR_print_errors(bio);
char *buf;
size_t len = BIO_get_mem_data(bio, &buf);
string ret(buf, len);
BIO_free(bio);
return ret;
}

*Kamala  Ayyar*
502 Claremont Ave.
Teaneck NJ 07666-2563
Tel: (201)530-0861


On Fri, Feb 4, 2022 at 10:54 AM Matt Caswell <matt at openssl.org> wrote:

>
>
> On 04/02/2022 15:17, Kamala Ayyar wrote:
> >
> > Hello,
> >
> > We are facing a strange handshake failure issue with a test server and
> > client application using OpenSSL in Windows.  We have tried with both
> > 1.1.1g and 3.0.1 versions- same problem. We created a Dll to handle the
> > OpenSSL functions- where the SSL context, SSL object and certificates
> > are handled. The certificates are obtained from the Windows store and
> > converted to cert and key using PKCS12_parse()
> > The server accepts non secure connection from the client and then passes
> > the socket to the Dll that calls the TLS_server_method() and creates the
> > SSL context, SSL object and loads the certificates for use. It however
> > fails at SSL_accept(m_pSsl). We use a call
> > back SSL_set_info_callback(m_pSsl, apps_ssl_info_callback) that gave us
> > the following error information
> > SSL_accept:Error in before SSL initialization
> > On the client side the same Dll is called with a client
> > method TLS_client_method() and the error displayed is SSL_connect:Error
> > in SSLv3/TLS write client hello
> > We have confirmed the certificates are good and valid.
> >
> > The same Dll called from a different heavily threaded application with
> > over 2000+ clients works well and handshake connections established
> > without issues on a different port number.
> >
> > We have also tried to use OpenSSL methods directly without using the Dll
> > but we get the same failure.  This was also used with server and client
> > on the same machine as well as different machines with the same
> > outcome.  The non secure communication works fine between the server and
> > the client
>
> What does SSL_get_error() report after SSL_accept() fails?
>
> Also please dump the OpenSSL error stack when it fails, e.g. using
> something like ERR_print_errors_fp(stdout);
>
> Matt
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mta.openssl.org/pipermail/openssl-users/attachments/20220204/2d2c17d4/attachment.htm>