Handshake Failure : SSL_accept:Error in before SSL initialization
Kamala Ayyar
kamala.ayyar at gmail.com
Tue Feb 8 18:17:57 UTC 2022
Hello Matt,
The socket descriptor is good and I verified using the socket directly. I
do exchange data between client and server successfully before passing it
the function to convert to a secure socket.
It fails at the same place as the SSL_accept() with the same error from
the call back function
*SSL_acceptSSL_CB_LOOP | SSL_accept:before SSL initializationSSL_accept*
*SSL_accept:Error in before SSL initialization*
On the client side the error is
*SSL_connectSSL_CB_LOOP | SSL_connect:before SSL
initializationSSL_connectSSL_connect:Error in SSLv3/TLS write client hello*
I used your example and wrapped the socket with the bio however I was not
able to do a bio read(). I am getting a return of -1 from the bio_read and
SSL_get_error() and ERR_prints_error does not print anything
Thanks
Kamala
On Fri, Feb 4, 2022 at 1:20 PM Matt Caswell <matt at openssl.org> wrote:
> Are you sure that the socket descriptor in "*this" is good and works?
>
> You could test that by wrapping it in a BIO like this:
>
> BIO *bio = BIO_new(BIO_s_socket());
>
> if (bio == NULL)
> goto err;
> BIO_set_fd(bio, *this, BIO_NOCLOSE);
>
> and then attempting to read some data from it using BIO_read(). If the
> BIO_read call fails then it suggests the socket descriptor is bad.
>
> Matt
>
>
>
> On 04/02/2022 18:06, Kamala Ayyar wrote:
> > Hello Matt,
> >
> > I call the WSAGetLastError() for Windows and that returns 183
> > (ERROR_ALREADY_EXISTS) //Cannot create a file when that file already
> exists
> > The SSL_get_error() gives us SSL_ERROR_SYSCALL
> > *Server *code is roughly like below
> > SSL_CTX *m_pCtx;
> > SSL *m_pSsl;
> > m_pCtx = SSL_CTX_new(TLS_server_method();
> > if ((dwRet = LoadCertificates()) != rSUCCESS)
> > throw dwRet;
> > if ((m_pSsl = SSL_new(m_pCtx)) != NULL)
> > {
> > if ((iRet = SSL_set_fd(m_pSsl, (*this)())) == 0) /* attach the
> > socket descriptor */
> > {
> > sslError = SSL_get_error(m_pSsl, iRet);
> > LOGERROR(szLine);
> > throw eSSL_ERROR;
> > }
> > SSL_set_info_callback(m_pSsl, apps_ssl_info_callback);
> > ERR_clear_error();
> > if ((sslError = SSL_accept(m_pSsl)) < 1)
> > {
> > sslError = SSL_get_error(m_pSsl, sslError);
> > dwRet = handleError(sslError, "SSL_accept failed with error ",
> > iRet);
> > throw dwRet;// eSSL_ERROR;
> > }
> > }
> >
> > Client
> > SSL_CTX *m_pCtx;
> > SSL *m_pSsl;
> > m_pCtx = SSL_CTX_new(TLS_client_method();
> > if ((dwRet = LoadCertificates(TRUE)) != rSUCCESS) //Trust certificates
> only
> > throw dwRet;
> > /* Set for server verification*/
> > SSL_CTX_set_verify(m_pCtx, SSL_VERIFY_PEER, NULL); //Work in progress
> > m_pSsl = SSL_new(m_pCtx);
> > if ((iRet = SSL_set_fd(m_pSsl, (*this)())) == 0) /* attach the socket
> > descriptor */
> > {
> > ssl_error = SSL_get_error(m_pSsl, iRet);
> > LOGERROR(szLine);
> > throw eSSL_ERROR;
> > }
> > SSL_set_info_callback(m_pSsl, apps_ssl_info_callback);
> > ERR_clear_error();
> > if ((iRet = SSL_connect(m_pSsl)) <= 0) /* perform the connection */
> > {
> > ssl_error = SSL_get_error(m_pSsl, iRet);
> > dwRet = handleError(iRet, "SSL_connect failed with error ", ssl_error);
> > throw eSSL_ERROR;
> > }
> >
> > ShowCerts();
> > }
> >
> > As mentioned before this code works fine when called by another
> > application. So the certificates are all valid. I also tried this on
> > different machines but it did not work- I get the same error.
> > Thanks
> > Kamala
> >
> > On Fri, Feb 4, 2022 at 12:20 PM Matt Caswell <matt at openssl.org
> > <mailto:matt at openssl.org>> wrote:
> >
> > Does errno give you anything?
> >
> > How did you create your BIOs for m_pSsl?
> >
> > Matt
> >
> > On 04/02/2022 16:25, Kamala Ayyar wrote:
> > > Hello Matt,
> > >
> > > The SSL_get_error() returns 5(SSL_ERROR_SYSCALL) It does not print
> > > anything for this error, just an empty string.
> > > I use the following to print error but nothing is printed
> > > if ((retVal = SSL_accept(m_pSsl)) < 1)
> > > {
> > > sslError = SSL_get_error(m_pSsl, retVal);
> > > LOGERROR(getOpenSSLError());
> > > throw dwRet;// eSSL_ERROR;
> > > }
> > > string getOpenSSLError()
> > > {
> > > BIO *bio = BIO_new(BIO_s_mem());
> > > ERR_print_errors(bio);
> > > char *buf;
> > > size_t len = BIO_get_mem_data(bio, &buf);
> > > string ret(buf, len);
> > > BIO_free(bio);
> > > return ret;
> > > }
> > >
> > > *Kamala Ayyar*
> > > 502 Claremont Ave.
> > > Teaneck NJ 07666-2563
> > > Tel: (201)530-0861
> > >
> > >
> > > On Fri, Feb 4, 2022 at 10:54 AM Matt Caswell <matt at openssl.org
> > <mailto:matt at openssl.org>
> > > <mailto:matt at openssl.org <mailto:matt at openssl.org>>> wrote:
> > >
> > >
> > >
> > > On 04/02/2022 15:17, Kamala Ayyar wrote:
> > > >
> > > > Hello,
> > > >
> > > > We are facing a strange handshake failure issue with a test
> > > server and
> > > > client application using OpenSSL in Windows. We have
> > tried with
> > > both
> > > > 1.1.1g and 3.0.1 versions- same problem. We created a Dll
> to
> > > handle the
> > > > OpenSSL functions- where the SSL context, SSL object and
> > > certificates
> > > > are handled. The certificates are obtained from the
> > Windows store
> > > and
> > > > converted to cert and key using PKCS12_parse()
> > > > The server accepts non secure connection from the client
> > and then
> > > passes
> > > > the socket to the Dll that calls the TLS_server_method()
> and
> > > creates the
> > > > SSL context, SSL object and loads the certificates for
> use. It
> > > however
> > > > fails at SSL_accept(m_pSsl). We use a call
> > > > back SSL_set_info_callback(m_pSsl, apps_ssl_info_callback)
> > that
> > > gave us
> > > > the following error information
> > > > SSL_accept:Error in before SSL initialization
> > > > On the client side the same Dll is called with a client
> > > > method TLS_client_method() and the error displayed
> > > is SSL_connect:Error
> > > > in SSLv3/TLS write client hello
> > > > We have confirmed the certificates are good and valid.
> > > >
> > > > The same Dll called from a different heavily threaded
> > application
> > > with
> > > > over 2000+ clients works well and handshake connections
> > established
> > > > without issues on a different port number.
> > > >
> > > > We have also tried to use OpenSSL methods directly
> > without using
> > > the Dll
> > > > but we get the same failure. This was also used with
> > server and
> > > client
> > > > on the same machine as well as different machines with the
> > same
> > > > outcome. The non secure communication works fine between
> the
> > > server and
> > > > the client
> > >
> > > What does SSL_get_error() report after SSL_accept() fails?
> > >
> > > Also please dump the OpenSSL error stack when it fails, e.g.
> > using
> > > something like ERR_print_errors_fp(stdout);
> > >
> > > Matt
> > >
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mta.openssl.org/pipermail/openssl-users/attachments/20220208/455c9cdb/attachment.htm>
More information about the openssl-users
mailing list