[openssl-users] How to handle DTLS Certificate Reassembly Error

[Matt Caswell]

On 16/09/16 19:47, Chad Phillips wrote:
> I’m using a support library leveraging openssl to complete a DTLS handshake.

You don't say what version of OpenSSL.

The packet trace you sent is quite confusing, as there appears to be two
separate handshakes going on at the same time that are interleaved. They
are both essentially the same though, so I filtered on just one of them
(the one involving UDP port 30041 - either as a source or destination
port). The analysis below is just on that one handshake.

> 
> Occasionally, I’ll see in my packet captures that a handshake has failed
> with a “Certificate reassembly error”, and the support library doesn’t
> seem to be catching this properly to forward the error on.

The "Certificate reassembly error" isn't really an error at all. This
appears to simply be a packet that has been retransmitted. In wireshark
if you select that packet and open up "Datagram Transport Layer
Security" -> "Record Layer" -> "Handshake Protocol", you will see that
this is an epoch 0 message (i.e. initial handshake message), with
message sequence 1 (the second message sent from the Client - the first
one being a ClientHello with message sequence 0). This contains the
first 231 bytes of the full Certificate - which is 469 bytes in length.

If you now compare this with the very first Certificate fragment sent
from client to server you will see that it is identical.

It seems quite clear that this is a retransmit of the earlier message
from client to server. Retransmits are a normal part of DTLS and are
there to handle packet loss. If a retransmitted packet is received by
one of the peers, and it has seen that packet before, then it is simply
ignored. Wireshark isn't ignoring it, and is reporting it as an "error"
simply because it has seen it before.

> 
> The library developers are considering handling this using a timeout
> solution — triggering an error if the handshake doesn’t complete in a
> specified amount of time, but this feels a bit clunky to me. What’s the
> recommended way to get this information from openssl in this case?
> 
> For reference, I’m attaching a packet capture that illustrates two of
> the handshake failures.

Was this packet capture done on the client side or the server side or
somewhere in the middle? There appears to be some messages missing. In
particular I don't see any CCS or Finished messages being exchanged. Is
the network this is over potentially noisy that might explain packet loss?

It is particularly odd because an epoch 1 encrypted alert can be seen to
have been sent from the server to the client which suggests that a CCS
*has* been sent from the client and received by the server - but it does
not appear in the packet trace.

Another odd thing is that it can clearly be seen that the server
retransmits the
ServerHello/Certificate/ServerKeyExchange/CertRequest/ServerHelloDone
flight. On receiving that the client should respond with a retransmit of
the Certificate/ClientKeyExchange/CertificateVerify/CCS/Finished flight
of messages. But it does not appear to do so...the retransmit does not
happen until after the encrypted alert.

Anyway, this really is a failed handshake, which for some reason I'm not
seeing the retransmits I would expect to see in order to make it
succeed. There has been no visible error, both peers are just sitting
there waiting for the other one to do something. That's not going to
happen if the the retransmits aren't working properly.

Are both ends of the communication using OpenSSL and if so what versions?

Matt