Handshake Failure : SSL_accept:Error in before SSL initialization
Matt Caswell
matt at openssl.org
Fri Feb 4 18:20:03 UTC 2022
Are you sure that the socket descriptor in "*this" is good and works?
You could test that by wrapping it in a BIO like this:
BIO *bio = BIO_new(BIO_s_socket());
if (bio == NULL)
goto err;
BIO_set_fd(bio, *this, BIO_NOCLOSE);
and then attempting to read some data from it using BIO_read(). If the
BIO_read call fails then it suggests the socket descriptor is bad.
Matt
On 04/02/2022 18:06, Kamala Ayyar wrote:
> Hello Matt,
>
> I call the WSAGetLastError() for Windows and that returns 183
> (ERROR_ALREADY_EXISTS) //Cannot create a file when that file already exists
> The SSL_get_error() gives us SSL_ERROR_SYSCALL
> *Server *code is roughly like below
> SSL_CTX *m_pCtx;
> SSL *m_pSsl;
> m_pCtx = SSL_CTX_new(TLS_server_method();
> if ((dwRet = LoadCertificates()) != rSUCCESS)
> throw dwRet;
> if ((m_pSsl = SSL_new(m_pCtx)) != NULL)
> {
> if ((iRet = SSL_set_fd(m_pSsl, (*this)())) == 0) /* attach the
> socket descriptor */
> {
> sslError = SSL_get_error(m_pSsl, iRet);
> LOGERROR(szLine);
> throw eSSL_ERROR;
> }
> SSL_set_info_callback(m_pSsl, apps_ssl_info_callback);
> ERR_clear_error();
> if ((sslError = SSL_accept(m_pSsl)) < 1)
> {
> sslError = SSL_get_error(m_pSsl, sslError);
> dwRet = handleError(sslError, "SSL_accept failed with error ",
> iRet);
> throw dwRet;// eSSL_ERROR;
> }
> }
>
> Client
> SSL_CTX *m_pCtx;
> SSL *m_pSsl;
> m_pCtx = SSL_CTX_new(TLS_client_method();
> if ((dwRet = LoadCertificates(TRUE)) != rSUCCESS) //Trust certificates only
> throw dwRet;
> /* Set for server verification*/
> SSL_CTX_set_verify(m_pCtx, SSL_VERIFY_PEER, NULL); //Work in progress
> m_pSsl = SSL_new(m_pCtx);
> if ((iRet = SSL_set_fd(m_pSsl, (*this)())) == 0) /* attach the socket
> descriptor */
> {
> ssl_error = SSL_get_error(m_pSsl, iRet);
> LOGERROR(szLine);
> throw eSSL_ERROR;
> }
> SSL_set_info_callback(m_pSsl, apps_ssl_info_callback);
> ERR_clear_error();
> if ((iRet = SSL_connect(m_pSsl)) <= 0) /* perform the connection */
> {
> ssl_error = SSL_get_error(m_pSsl, iRet);
> dwRet = handleError(iRet, "SSL_connect failed with error ", ssl_error);
> throw eSSL_ERROR;
> }
>
> ShowCerts();
> }
>
> As mentioned before this code works fine when called by another
> application. So the certificates are all valid. I also tried this on
> different machines but it did not work- I get the same error.
> Thanks
> Kamala
>
> On Fri, Feb 4, 2022 at 12:20 PM Matt Caswell <matt at openssl.org
> <mailto:matt at openssl.org>> wrote:
>
> Does errno give you anything?
>
> How did you create your BIOs for m_pSsl?
>
> Matt
>
> On 04/02/2022 16:25, Kamala Ayyar wrote:
> > Hello Matt,
> >
> > The SSL_get_error() returns 5(SSL_ERROR_SYSCALL) It does not print
> > anything for this error, just an empty string.
> > I use the following to print error but nothing is printed
> > if ((retVal = SSL_accept(m_pSsl)) < 1)
> > {
> > sslError = SSL_get_error(m_pSsl, retVal);
> > LOGERROR(getOpenSSLError());
> > throw dwRet;// eSSL_ERROR;
> > }
> > string getOpenSSLError()
> > {
> > BIO *bio = BIO_new(BIO_s_mem());
> > ERR_print_errors(bio);
> > char *buf;
> > size_t len = BIO_get_mem_data(bio, &buf);
> > string ret(buf, len);
> > BIO_free(bio);
> > return ret;
> > }
> >
> > *Kamala Ayyar*
> > 502 Claremont Ave.
> > Teaneck NJ 07666-2563
> > Tel: (201)530-0861
> >
> >
> > On Fri, Feb 4, 2022 at 10:54 AM Matt Caswell <matt at openssl.org
> <mailto:matt at openssl.org>
> > <mailto:matt at openssl.org <mailto:matt at openssl.org>>> wrote:
> >
> >
> >
> > On 04/02/2022 15:17, Kamala Ayyar wrote:
> > >
> > > Hello,
> > >
> > > We are facing a strange handshake failure issue with a test
> > server and
> > > client application using OpenSSL in Windows. We have
> tried with
> > both
> > > 1.1.1g and 3.0.1 versions- same problem. We created a Dll to
> > handle the
> > > OpenSSL functions- where the SSL context, SSL object and
> > certificates
> > > are handled. The certificates are obtained from the
> Windows store
> > and
> > > converted to cert and key using PKCS12_parse()
> > > The server accepts non secure connection from the client
> and then
> > passes
> > > the socket to the Dll that calls the TLS_server_method() and
> > creates the
> > > SSL context, SSL object and loads the certificates for use. It
> > however
> > > fails at SSL_accept(m_pSsl). We use a call
> > > back SSL_set_info_callback(m_pSsl, apps_ssl_info_callback)
> that
> > gave us
> > > the following error information
> > > SSL_accept:Error in before SSL initialization
> > > On the client side the same Dll is called with a client
> > > method TLS_client_method() and the error displayed
> > is SSL_connect:Error
> > > in SSLv3/TLS write client hello
> > > We have confirmed the certificates are good and valid.
> > >
> > > The same Dll called from a different heavily threaded
> application
> > with
> > > over 2000+ clients works well and handshake connections
> established
> > > without issues on a different port number.
> > >
> > > We have also tried to use OpenSSL methods directly
> without using
> > the Dll
> > > but we get the same failure. This was also used with
> server and
> > client
> > > on the same machine as well as different machines with the
> same
> > > outcome. The non secure communication works fine between the
> > server and
> > > the client
> >
> > What does SSL_get_error() report after SSL_accept() fails?
> >
> > Also please dump the OpenSSL error stack when it fails, e.g.
> using
> > something like ERR_print_errors_fp(stdout);
> >
> > Matt
> >
>
More information about the openssl-users
mailing list