[openssl-users] OpenSSL handshake failure in ssl3_get_client_hello() routine

[Senthil Raja Velu]

Hi,
I have a setup where the handshake between openssl server and client fails
at times but not always. And when it does,  the client keeps retrying and
all of trials fail. Only way to recover is to restart the server.

Currently on the server side the openssl version that I have installed is
1.0.1m.

Both server and client are written in C and are in non-blocking mode. I
have added InfoCallBack and printCallBack routines on the server side.

On the server side application, I have set the following options;

    pCtx = SSL_CTX_new(SSLv23_server_method());

    SSL_CTX_set_options(pCtx, SSL_OP_NO_SSLv2);
    SSL_CTX_set_options(pCtx, SSL_OP_NO_SSLv3);
    SSL_CTX_set_options(pCtx, SSL_OP_NO_TLSv1);


Now when the failure occurs, I get the following error message on the
server side:

InfoCB
HANDSHAKE_START(time:5093879)  undefined: before/accept initialization

InfoCB
SSL_accept:before/accept initialization

InfoCB
SSL3 alert write:fatal:internal error

PrintCB
error:1408A044:SSL routines:SSL3_GET_CLIENT_HELLO:internal
error:/server/openssl/ssl/s3_srvr.c:1265:

InfoCB
SSL_accept:error in SSLv3 read client hello C

InfoCB
SSL_accept:error in SSLv3 read client hello C


The SSL code path </server/openssl/ssl/s3_srvr.c:1265> refers to the
following section of code in ssl3_get_client_hello() routine in s3_srvr.c.

--------------------------------------------------------------------------
    /*
     * Check if we want to use external pre-shared secret for this handshake
     * for not reused session only. We need to generate server_random before
     * calling tls_session_secret_cb in order to allow SessionTicket
     * processing to use it in key derivation.
     */
    {
        unsigned char *pos;
        pos = s->s3->server_random;
        if (ssl_fill_hello_random(s, 1, pos, SSL3_RANDOM_SIZE) <= 0) {
#ifdef USER_EXTENSIONS
            SSLerr(SSL_F_SSL3_GET_CLIENT_HELLO, ERR_R_INTERNAL_ERROR);
#endif // USER_EXTENSIONS
            goto f_err;
        }
    }
--------------------------------------------------------------------------

Note, I have edited the SSL library to include this USER_EXTENSIONS
section, so that I could confirm where exactly this issue is happening in
the library.

Clearly ssl_fill_hello_ramdom() routine is returning -1 or something less
than zero.

I do not hit this issue always.

Any pointers on addressing this issue will be a big help.


Thanks,
Senthil.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mta.openssl.org/pipermail/openssl-users/attachments/20170126/a97b2c9f/attachment.html>