[openssl-users] Query regarding DTLS handshake

[Matt Caswell]

On 13/04/17 10:11, mahesh gs wrote:
> Hi,
> 
> We are running SCTP connections with DTLS enabled in our application. We
> have adapted openssl version (openssl-1.1.0e) to achieve the same.
> 
> We have generated the self signed root and node certificates for
> testing. We have a strange problem with the incomplete DTLS handshake if
> we run the DTLS client and DTLS server is different systems.If we run
> the DTLS client and server in same system handshake is successful,
> handshake is not successful if run client and server in different VM's.
> 
> This strange problem happens only for SCTP/DTLS connection. With the
> same set of certificates TCP/TLS connection is successful and we are
> able to exchange the application data.
> 
> I am attaching the code bits for SSL_accept and SSL_connect and also the
> wireshark trace of unsuccessful handshake. Please assist me to debug
> this problem.
> 
> SSL_accept returns  SSL_ERROR_WANT_READ(2) infinite times but
> SSL_connect is called 4 or 5 times and select system call timeout.

Your trace shows the following interactions occurring:

Client                         Server
------                         ------

ClientHello          -------->
                     <-------- ServerHello
                     <-------- Certificate
                     <-------- CertificateRequest
                     <-------- ServerDone
Certificate          --------->
ClientKeyExchange    --------->
CertificateVerify    --------->
CCS                  --------->
[Encrypted Finished]

We would expect the server to continue with its own CCS and Encrypted
Finished to complete the handshake. It seems that, for some reason, the
server is not receiving (or acting upon) the client's second flight of
messages.

Normally in DTLS this sort of thing can happen due to lost messages etc
but, obviously, with SCTP, this is not the case. Something else must be
happening.

In your description you say SSL_accept() gets called repeatedly and
always gives SSL_ERROR_WANT_READ. Looking at your code it looks like you
are calling pollSocketForEvents() after each accept. I am assuming that
this is returning true each time (otherwise you would break out of the
loop). This suggests that the "select" call thinks there is something to
read from the underlying socket. Am I correct? The question is why
doesn't OpenSSL then read that data out of the socket?

Are you able to build a debug version of OpenSSL (run "config" with -d),
and step through to figure out where it gets stuck. Is it attempting to
read the data and failing, or does it not get as far attempting to read it?

Another question: does this fail every time or does it sometimes work
and sometimes not (which might suggest some race condition)?

Matt