[openssl-users] Trying to understand DTLS (as it applies to webrtc)

[Matt Caswell]

On 01/05/15 02:11, faraz khan wrote:
> Hi everyone,
> This is my first time posting to this list - so if theres a better place
> for this question please let me know.
> 
> The problem I'm trying to fix applies to the Janus webrtc gateway
> (https://github.com/meetecho/janus-gateway) and my application which is
> using native C++ webrtc.
> 
> What happens is that after hundreds of successful connections, sometimes
> the Janus server is unable to negotiate a DTLS handshake and after a key
> exchange the webrtc client replied with a DTLS Alert: Decrypt failed
> message. I'm attaching a wireshark trace of the issue happening and one
> for the correct negotiation.  
> 
> The problem refuses to fix itself till Janus is restarted.
> 
> Both installations are using Openssl. Janus is compiled with version 1.0.1f
> 
> If someone can help explain how DTLS key exchange works and whats going
> wrong in the above trace it would be great! I'm completely at a loss as
> far as this is concerned!
> 
> Thanks all!

Hmmmm. I can't see anything obviously wrong with the above traces. The
handshake seems to proceed as normal and then fail near the end.

A couple of things of note:
* A client cert is being sent, but it has expired. I don't think this is
the problem though because it is the same cert in the "good" trace and
the "bad" trace.
        Validity
            Not Before: Feb  9 16:18:45 2007 GMT
            Not After : Feb  8 16:18:45 2009 GMT

* A different ciphersuite is being negotiated between the "good" version
and the "bad" version. "Good" is TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,
whilst "Bad" is TLS_RSA_WITH_AES_256_CBC_SHA. I'm not sure if that is
significant, but I can't see why a server restart would make any
difference if it were.

Are there any server logs which might indicate why it is sending the
alert? Looking at the code there are only a few places in the code which
generate a decrypt error alert. It would probably help diagnose the
problem if we could narrow down which of those places this is coming
from. OpenSSL adds an error to its error queue for each of those places.

The other point of note is that there have been quite a lot of DTLS
related defect fixes in the OpenSSL code since 1.0.1f. An upgrade would
be a really good idea.

Matt