[openssl-users] DTLS Handshake fails with DTLSv1_listen

[Matt Caswell]

On 23/02/17 18:02, Vijayakumar Kaliaperumal wrote:
> Hi, 
> 
> While writing  a DTLS server using  DTLSv1_listen(),   I found  that 
> when I receive a fragmented clienthello from the client,  DTLS handshake
> fails.  DTLSv1_listen stuck in the while loop (in the app).
> When I checked the man page of DTLSv1_listen(),  it clearly says that
> API does not handle a fragmented clienthello.  as it operates entirely
> statelessly ( Safeguard against  DOS attacks ? )

Yes, DTLS is particularly susceptible to DoS attacks so the whole point
of DTLSv1_listen() is to avoid that.

> 
> However DTLS RFC clearly states that implementation must handle
> fragmented handshake messages
> 
> RFC 4347 Datagram Transport Layer Security:
> “When a DTLS implementation receives a handshake message fragment, it
> MUST buffer it until it has the entire handshake message.”
> 

Yes this issue was the subject of some discussion when we rewrote the
DTLSv1_listen() implementation for version 1.1.0.

Actually the documentation is slightly out-of-date. It was true at the
time we did the initial rewrite, but we later made some amendments to
relax that requirement a little. We now allow fragmented ClientHellos as
long as the cookie is fully contained within the first fragment. We
should probably fix the docs to say that. Fragmenting a ClientHello so
much that that property isn't true would be a fairly mad thing to do IMO.

You don't say which OpenSSL version you are using - 1.0.2 or 1.1.0. If
1.0.2 I would highly recommend that you upgrade to 1.1.0. The 1.0.2
implementation has all sorts of problems that are just unfixable (which
is why we rewrote it).

If you're already using 1.1.0, are you encountering ClientHello's that
are fragmented to the point that the cookie isn't in the first
fragment?? If so I'd be interested to know what client is doing that!


> Avoiding the fragmented clienthello is the only way out for this problem
> ? or any other alternatives exist ?

There are 3 ways of doing this with OpenSSL:

1) Fully stateless. This is the preferred way - using DTLSv1_listen()

2) Stateful with cookie. With this way of doing things you don't use
DTLSv1_listen() at all. You just use SSL_accept() like you would for a
TLS connection. The underlying BIO is expected to be in a "connected"
state. You need to set the SSL_OP_COOKIE_EXCHANGE option. This gives you
"some" protection from DoS. It still allocates state on the server but
some of the more expensive operations are deferred until after the
cookie exchange loop has been completed.

3) Stateful without cookie. Like (2) but don't set
SSL_OP_COOKIE_EXCHANGE. No protection at all from DoS.

Approaches 2 + 3 can handle fully fragmented ClientHellos at the expense
of reduced/no protection from DoS.

Matt