Handshake Failure : SSL_accept:Error in before SSL initialization

Kamala Ayyar kamala.ayyar at gmail.com
Tue Feb 8 18:17:57 UTC 2022


Hello Matt,

The socket descriptor is good and I verified using the socket directly.  I
do exchange data between client and server successfully before passing it
the function to convert to a secure socket.
It fails at the same place as the SSL_accept() with the same error  from
the call back function




*SSL_acceptSSL_CB_LOOP | SSL_accept:before SSL initializationSSL_accept*
*SSL_accept:Error in before SSL initialization*
On the client side the error is




*SSL_connectSSL_CB_LOOP | SSL_connect:before SSL
initializationSSL_connectSSL_connect:Error in SSLv3/TLS write client hello*
I used your example and wrapped the socket with the bio however I was not
able to do a bio read(). I am getting a return of -1 from the bio_read and
SSL_get_error() and ERR_prints_error does not print anything

Thanks
Kamala



On Fri, Feb 4, 2022 at 1:20 PM Matt Caswell <matt at openssl.org> wrote:

> Are you sure that the socket descriptor in "*this" is good and works?
>
> You could test that by wrapping it in a BIO like this:
>
>      BIO *bio = BIO_new(BIO_s_socket());
>
>      if (bio == NULL)
>         goto err;
>      BIO_set_fd(bio, *this, BIO_NOCLOSE);
>
> and then attempting to read some data from it using BIO_read(). If the
> BIO_read call fails then it suggests the socket descriptor is bad.
>
> Matt
>
>
>
> On 04/02/2022 18:06, Kamala Ayyar wrote:
> > Hello Matt,
> >
> > I call the WSAGetLastError() for Windows and that returns 183
> > (ERROR_ALREADY_EXISTS) //Cannot create a file when that file already
> exists
> > The SSL_get_error() gives us SSL_ERROR_SYSCALL
> > *Server *code is roughly like below
> > SSL_CTX *m_pCtx;
> > SSL *m_pSsl;
> > m_pCtx = SSL_CTX_new(TLS_server_method();
> > if ((dwRet = LoadCertificates()) != rSUCCESS)
> > throw dwRet;
> > if ((m_pSsl = SSL_new(m_pCtx)) != NULL)
> > {
> >       if ((iRet = SSL_set_fd(m_pSsl, (*this)())) == 0) /* attach the
> > socket descriptor */
> >      {
> >      sslError = SSL_get_error(m_pSsl, iRet);
> >      LOGERROR(szLine);
> >      throw eSSL_ERROR;
> >      }
> >     SSL_set_info_callback(m_pSsl, apps_ssl_info_callback);
> >     ERR_clear_error();
> >     if ((sslError = SSL_accept(m_pSsl)) < 1)
> >       {
> >           sslError = SSL_get_error(m_pSsl, sslError);
> >           dwRet = handleError(sslError, "SSL_accept failed with error ",
> > iRet);
> >           throw dwRet;// eSSL_ERROR;
> >       }
> > }
> >
> > Client
> > SSL_CTX *m_pCtx;
> > SSL *m_pSsl;
> > m_pCtx = SSL_CTX_new(TLS_client_method();
> > if ((dwRet = LoadCertificates(TRUE)) != rSUCCESS) //Trust certificates
> only
> > throw dwRet;
> > /* Set for server verification*/
> > SSL_CTX_set_verify(m_pCtx, SSL_VERIFY_PEER, NULL); //Work in progress
> > m_pSsl = SSL_new(m_pCtx);
> > if ((iRet = SSL_set_fd(m_pSsl, (*this)())) == 0) /* attach the socket
> > descriptor */
> > {
> >     ssl_error = SSL_get_error(m_pSsl, iRet);
> >     LOGERROR(szLine);
> >     throw eSSL_ERROR;
> > }
> > SSL_set_info_callback(m_pSsl, apps_ssl_info_callback);
> > ERR_clear_error();
> > if ((iRet = SSL_connect(m_pSsl)) <= 0)   /* perform the connection */
> > {
> > ssl_error = SSL_get_error(m_pSsl, iRet);
> > dwRet = handleError(iRet, "SSL_connect failed with error ", ssl_error);
> > throw eSSL_ERROR;
> > }
> >
> > ShowCerts();
> > }
> >
> > As mentioned before this code works fine when  called by another
> > application. So the certificates are all valid. I also tried this on
> > different machines  but it did not work- I get the same error.
> > Thanks
> > Kamala
> >
> > On Fri, Feb 4, 2022 at 12:20 PM Matt Caswell <matt at openssl.org
> > <mailto:matt at openssl.org>> wrote:
> >
> >     Does errno give you anything?
> >
> >     How did you create your BIOs for m_pSsl?
> >
> >     Matt
> >
> >     On 04/02/2022 16:25, Kamala Ayyar wrote:
> >      > Hello Matt,
> >      >
> >      > The SSL_get_error() returns 5(SSL_ERROR_SYSCALL) It does not print
> >      > anything for this error, just an empty string.
> >      > I use the following to print error but nothing is printed
> >      > if ((retVal = SSL_accept(m_pSsl)) < 1)
> >      > {
> >      > sslError = SSL_get_error(m_pSsl, retVal);
> >      > LOGERROR(getOpenSSLError());
> >      > throw dwRet;// eSSL_ERROR;
> >      > }
> >      > string getOpenSSLError()
> >      > {
> >      > BIO *bio = BIO_new(BIO_s_mem());
> >      > ERR_print_errors(bio);
> >      > char *buf;
> >      > size_t len = BIO_get_mem_data(bio, &buf);
> >      > string ret(buf, len);
> >      > BIO_free(bio);
> >      > return ret;
> >      > }
> >      >
> >      > *Kamala  Ayyar*
> >      > 502 Claremont Ave.
> >      > Teaneck NJ 07666-2563
> >      > Tel: (201)530-0861
> >      >
> >      >
> >      > On Fri, Feb 4, 2022 at 10:54 AM Matt Caswell <matt at openssl.org
> >     <mailto:matt at openssl.org>
> >      > <mailto:matt at openssl.org <mailto:matt at openssl.org>>> wrote:
> >      >
> >      >
> >      >
> >      >     On 04/02/2022 15:17, Kamala Ayyar wrote:
> >      >      >
> >      >      > Hello,
> >      >      >
> >      >      > We are facing a strange handshake failure issue with a test
> >      >     server and
> >      >      > client application using OpenSSL in Windows.  We have
> >     tried with
> >      >     both
> >      >      > 1.1.1g and 3.0.1 versions- same problem. We created a Dll
> to
> >      >     handle the
> >      >      > OpenSSL functions- where the SSL context, SSL object and
> >      >     certificates
> >      >      > are handled. The certificates are obtained from the
> >     Windows store
> >      >     and
> >      >      > converted to cert and key using PKCS12_parse()
> >      >      > The server accepts non secure connection from the client
> >     and then
> >      >     passes
> >      >      > the socket to the Dll that calls the TLS_server_method()
> and
> >      >     creates the
> >      >      > SSL context, SSL object and loads the certificates for
> use. It
> >      >     however
> >      >      > fails at SSL_accept(m_pSsl). We use a call
> >      >      > back SSL_set_info_callback(m_pSsl, apps_ssl_info_callback)
> >     that
> >      >     gave us
> >      >      > the following error information
> >      >      > SSL_accept:Error in before SSL initialization
> >      >      > On the client side the same Dll is called with a client
> >      >      > method TLS_client_method() and the error displayed
> >      >     is SSL_connect:Error
> >      >      > in SSLv3/TLS write client hello
> >      >      > We have confirmed the certificates are good and valid.
> >      >      >
> >      >      > The same Dll called from a different heavily threaded
> >     application
> >      >     with
> >      >      > over 2000+ clients works well and handshake connections
> >     established
> >      >      > without issues on a different port number.
> >      >      >
> >      >      > We have also tried to use OpenSSL methods directly
> >     without using
> >      >     the Dll
> >      >      > but we get the same failure.  This was also used with
> >     server and
> >      >     client
> >      >      > on the same machine as well as different machines with the
> >     same
> >      >      > outcome.  The non secure communication works fine between
> the
> >      >     server and
> >      >      > the client
> >      >
> >      >     What does SSL_get_error() report after SSL_accept() fails?
> >      >
> >      >     Also please dump the OpenSSL error stack when it fails, e.g.
> >     using
> >      >     something like ERR_print_errors_fp(stdout);
> >      >
> >      >     Matt
> >      >
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mta.openssl.org/pipermail/openssl-users/attachments/20220208/455c9cdb/attachment.htm>


More information about the openssl-users mailing list