[openssl-dev] [openssl.org #4119] DTLS resets handshake hash too frequently for ClientHello

[Matt Caswell via RT]

Hi David

On 03/11/15 01:58, David Benjamin via RT wrote:
> Hey folks,
> 
> We found a small DTLS bug while writing some tests. I think it affects
> 1.0.1 and 1.0.2 too, so I thought I'd send you a note. (Note sure about
> master. I'm unfamiliar with the new state machine mechanism.)
> 
> In DTLS, each ClientHello is supposed to reset the handshake hash (in case
> of HelloVerifyRequest). The old state machine calls ssl3_init_finished_mac
> in the SSL3_ST_CW_CLNT_HELLO_* states.
> 
> https://git.openssl.org/gitweb/?p=openssl.git;a=blob;f=ssl/d1_clnt.c;h=feeaf6d0656f5d0868121852d42b5037b8823111;hb=refs/heads/OpenSSL_1_0_2-stable#l317
> 
> If the ClientHello is fragmented and takes multiple iterations to write,
> the state machine is re-entered in SSL3_ST_CW_CLNT_HELLO_B, which
> calls ssl3_init_finished_mac again, and clobbers what was sampled to the
> handshake hash/buffer previously.
> 
> This requires the transport return a retriable error on write, which
> probably isn't common for DTLS. It came up because WebRTC uses a custom BIO
> with a fixed-size buffer, so it can actually do this. Even then, a
> ClientHello is unlikely to fill up the buffer. Our tests repro'd it in
> BoringSSL by forcing every write to take two iterations.

I can confirm this issue on 1.0.2 (and almost certainly 1.0.1). It does
not affect master.

Whilst investigating this I noticed another bug which is actually
probably more significant. My eyeball only look at the BoringSSL source
suggests that it is there too, so I'm not sure why you haven't seen it
in the test that you mentioned.

If a retry occurs on a second or subsequent fragment of a handshake
message then when we do the retry the wrong fragment offset and length
is used. The impact isn't that great because the messages got dropped by
the peer, and then when they get retransmitted they have the correct
values inserted...so the handshake succeeds - but it gets delayed.
Perhaps that is why you don't see it in your test.

This second issue occurs in all branches.

One other related point is that fragmenting ClientHellos is probably a
bad idea. The whole ClientHello/HelloVerifyRequest mechanism is meant to
be implemented without storing state on the server. That isn't possible
if you have to deal with fragment reassembly. In the new DTLSv1_listen
implementation in master we drop fragmented ClientHellos.

Patch attached for these two issues (patch against 1.0.2).

Matt


-------------- next part --------------
A non-text attachment was scrubbed...
Name: dtls-frag-retry.patch
Type: text/x-patch
Size: 3085 bytes
Desc: not available
URL: <http://mta.openssl.org/pipermail/openssl-dev/attachments/20151103/19cfdbfd/attachment.bin>