Handshake Failure : SSL_accept:Error in before SSL initialization

[Kamala Ayyar]

Hello Matt,

I call the WSAGetLastError() for Windows and that returns 183
(ERROR_ALREADY_EXISTS) //Cannot create a file when that file already exists
The SSL_get_error() gives us  SSL_ERROR_SYSCALL
*Server *code is roughly like below
SSL_CTX *m_pCtx;
SSL *m_pSsl;
m_pCtx = SSL_CTX_new(TLS_server_method();
if ((dwRet = LoadCertificates()) != rSUCCESS)
throw dwRet;
if ((m_pSsl = SSL_new(m_pCtx)) != NULL)
{
     if ((iRet = SSL_set_fd(m_pSsl, (*this)())) == 0) /* attach the socket
descriptor */
    {
    sslError = SSL_get_error(m_pSsl, iRet);
    LOGERROR(szLine);
    throw eSSL_ERROR;
    }
   SSL_set_info_callback(m_pSsl, apps_ssl_info_callback);
   ERR_clear_error();
   if ((sslError = SSL_accept(m_pSsl)) < 1)
     {
         sslError = SSL_get_error(m_pSsl, sslError);
         dwRet = handleError(sslError, "SSL_accept failed with error ",
iRet);
         throw dwRet;// eSSL_ERROR;
     }
}

Client
SSL_CTX *m_pCtx;
SSL *m_pSsl;
m_pCtx = SSL_CTX_new(TLS_client_method();
if ((dwRet = LoadCertificates(TRUE)) != rSUCCESS) //Trust certificates only
throw dwRet;
/* Set for server verification*/
SSL_CTX_set_verify(m_pCtx, SSL_VERIFY_PEER, NULL); //Work in progress
m_pSsl = SSL_new(m_pCtx);
if ((iRet = SSL_set_fd(m_pSsl, (*this)())) == 0) /* attach the socket
descriptor */
{
   ssl_error = SSL_get_error(m_pSsl, iRet);
   LOGERROR(szLine);
   throw eSSL_ERROR;
}
SSL_set_info_callback(m_pSsl, apps_ssl_info_callback);
ERR_clear_error();
if ((iRet = SSL_connect(m_pSsl)) <= 0)   /* perform the connection */
{
ssl_error = SSL_get_error(m_pSsl, iRet);
dwRet = handleError(iRet, "SSL_connect failed with error ", ssl_error);
throw eSSL_ERROR;
}

ShowCerts();
}

As mentioned before this code works fine when  called by another
application. So the certificates are all valid. I also tried this on
different machines  but it did not work- I get the same error.
Thanks
Kamala

On Fri, Feb 4, 2022 at 12:20 PM Matt Caswell <matt at openssl.org> wrote:

> Does errno give you anything?
>
> How did you create your BIOs for m_pSsl?
>
> Matt
>
> On 04/02/2022 16:25, Kamala Ayyar wrote:
> > Hello Matt,
> >
> > The SSL_get_error() returns 5(SSL_ERROR_SYSCALL) It does not print
> > anything for this error, just an empty string.
> > I use the following to print error but nothing is printed
> > if ((retVal = SSL_accept(m_pSsl)) < 1)
> > {
> > sslError = SSL_get_error(m_pSsl, retVal);
> > LOGERROR(getOpenSSLError());
> > throw dwRet;// eSSL_ERROR;
> > }
> > string getOpenSSLError()
> > {
> > BIO *bio = BIO_new(BIO_s_mem());
> > ERR_print_errors(bio);
> > char *buf;
> > size_t len = BIO_get_mem_data(bio, &buf);
> > string ret(buf, len);
> > BIO_free(bio);
> > return ret;
> > }
> >
> > *Kamala  Ayyar*
> > 502 Claremont Ave.
> > Teaneck NJ 07666-2563
> > Tel: (201)530-0861
> >
> >
> > On Fri, Feb 4, 2022 at 10:54 AM Matt Caswell <matt at openssl.org
> > <mailto:matt at openssl.org>> wrote:
> >
> >
> >
> >     On 04/02/2022 15:17, Kamala Ayyar wrote:
> >      >
> >      > Hello,
> >      >
> >      > We are facing a strange handshake failure issue with a test
> >     server and
> >      > client application using OpenSSL in Windows.  We have tried with
> >     both
> >      > 1.1.1g and 3.0.1 versions- same problem. We created a Dll to
> >     handle the
> >      > OpenSSL functions- where the SSL context, SSL object and
> >     certificates
> >      > are handled. The certificates are obtained from the Windows store
> >     and
> >      > converted to cert and key using PKCS12_parse()
> >      > The server accepts non secure connection from the client and then
> >     passes
> >      > the socket to the Dll that calls the TLS_server_method() and
> >     creates the
> >      > SSL context, SSL object and loads the certificates for use. It
> >     however
> >      > fails at SSL_accept(m_pSsl). We use a call
> >      > back SSL_set_info_callback(m_pSsl, apps_ssl_info_callback) that
> >     gave us
> >      > the following error information
> >      > SSL_accept:Error in before SSL initialization
> >      > On the client side the same Dll is called with a client
> >      > method TLS_client_method() and the error displayed
> >     is SSL_connect:Error
> >      > in SSLv3/TLS write client hello
> >      > We have confirmed the certificates are good and valid.
> >      >
> >      > The same Dll called from a different heavily threaded application
> >     with
> >      > over 2000+ clients works well and handshake connections
> established
> >      > without issues on a different port number.
> >      >
> >      > We have also tried to use OpenSSL methods directly without using
> >     the Dll
> >      > but we get the same failure.  This was also used with server and
> >     client
> >      > on the same machine as well as different machines with the same
> >      > outcome.  The non secure communication works fine between the
> >     server and
> >      > the client
> >
> >     What does SSL_get_error() report after SSL_accept() fails?
> >
> >     Also please dump the OpenSSL error stack when it fails, e.g. using
> >     something like ERR_print_errors_fp(stdout);
> >
> >     Matt
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mta.openssl.org/pipermail/openssl-users/attachments/20220204/2d98fbab/attachment.htm>