[openssl-users] OpenSSL handshake failure in ssl3_get_client_hello() routine

[Matt Caswell]

On 26/01/17 04:38, Senthil Raja Velu wrote:
> Hi,
> I have a setup where the handshake between openssl server and client
> fails at times but not always. And when it does,  the client keeps
> retrying and all of trials fail. Only way to recover is to restart the
> server.
> 
> Currently on the server side the openssl version that I have installed
> is 1.0.1m.

That's quite an old version and is likely to be vulnerable to various
security issues. You should upgrade. Further the 1.0.1 series is no
longer supported (unless your 1.0.1m is actually supplied by your OS
vendor - in which case they may be backporting security fixes to it). If
you are not using an OS supplied version then I recommend you upgrade to
version 1.0.2k (which should be a straight forward upgrade) or 1.1.0d
(which may be more difficult). Those versions will be released later today.

> The SSL code path </server/openssl/ssl/s3_srvr.c:1265> refers to the
> following section of code in ssl3_get_client_hello() routine in s3_srvr.c.
> 
> --------------------------------------------------------------------------
>     /*
>      * Check if we want to use external pre-shared secret for this handshake
>      * for not reused session only. We need to generate server_random before
>      * calling tls_session_secret_cb in order to allow SessionTicket
>      * processing to use it in key derivation.
>      */
>     {
>         unsigned char *pos;
>         pos = s->s3->server_random;
>         if (ssl_fill_hello_random(s, 1, pos, SSL3_RANDOM_SIZE) <= 0) {
> #ifdef USER_EXTENSIONS
>             SSLerr(SSL_F_SSL3_GET_CLIENT_HELLO, ERR_R_INTERNAL_ERROR);
> #endif // USER_EXTENSIONS
>             goto f_err;
>         }
>     }
> --------------------------------------------------------------------------
> 
> Note, I have edited the SSL library to include this USER_EXTENSIONS
> section, so that I could confirm where exactly this issue is happening
> in the library.
> 
> Clearly ssl_fill_hello_ramdom() routine is returning -1 or something
> less than zero.

Well zero or less to be exact. The code for ssl_fill_hello_random()
looks like this:

int ssl_fill_hello_random(SSL *s, int server, unsigned char *result, int
len)
{
    int send_time = 0;

    if (len < 4)
        return 0;
    if (server)
        send_time = (s->mode & SSL_MODE_SEND_SERVERHELLO_TIME) != 0;
    else
        send_time = (s->mode & SSL_MODE_SEND_CLIENTHELLO_TIME) != 0;
    if (send_time) {
        unsigned long Time = (unsigned long)time(NULL);
        unsigned char *p = result;
        l2n(Time, p);
        return RAND_pseudo_bytes(p, len - 4);
    } else
        return RAND_pseudo_bytes(result, len);
}


As you can see it can return 0 if len < 4 - but in this case it is clear
that that isn't happening (because len is set to SSL3_RANDOM_SIZE == 32).

Otherwise it returns the result of RAND_pseudo_bytes(). There are a few
reasons why that function returns <= 0:

1) It can't find the random method to use (either built-in or default).
This is really a "should never happen" type condition.

2) If using the default random method then it has insufficient entropy.

3) If using an engine supplied random method, then it has failed for
some engine specific reason.

Are you using an engine that might supply its own random method? If so
you might want to look at whether that is failing.

If not, then look here:
https://www.openssl.org/docs/faq.html#USER1

Incidentally if you were to do the upgrade to 1.0.2 or 1.1.0 then you
would probably get an additional error message confirming that it is a
low entropy issue. In 1.0.2 the RAND_pseudo_bytes() call has been
changed to RAND_bytes(). These two are very similar, but on failure due
to low entropy RAND_bytes() puts an error in the error queue.

Matt