[openssl-users] Trying to understand DTLS (as it applies to webrtc)

[Matt Caswell]

On 01/05/15 02:11, faraz khan wrote:
> Hi everyone,
> This is my first time posting to this list - so if theres a better place
> for this question please let me know.
> 
> The problem I'm trying to fix applies to the Janus webrtc gateway
> (https://github.com/meetecho/janus-gateway) and my application which is
> using native C++ webrtc.
> 
> What happens is that after hundreds of successful connections, sometimes
> the Janus server is unable to negotiate a DTLS handshake and after a key
> exchange the webrtc client replied with a DTLS Alert: Decrypt failed
> message. I'm attaching a wireshark trace of the issue happening and one
> for the correct negotiation.

Can you confirm that the trace you attached for the correct negotiation
also came from Janus? I spotted some odd things about the extensions in
the ClientHello:

It has the "extended master secret" extension. That extension is not
supported by any released version of OpenSSL (we have it in the dev
version of 1.1.0 - but that is unreleased).

It correctly adds the ec_point_formats and elliptic_curves extensions.
However it so happens that OpenSSL 1.0.1f has a bug in it (not in the
latest version) which suppresses those extensions for DTLS (this is a
significant enough bug in itself that it is worthwhile upgrading your
OpenSSL)

OpenSSL always adds extensions in the same order - and these extensions
are in a different order.

Due to the above it looks to me like the OpenSSL DTLS stack was not used
to generate that ClientHello. The failed one *does* look like an OpenSSL
generated ClientHello.

Matt