[openssl-users] Trying to understand DTLS (as it applies to webrtc)

[Matt Caswell]

On 01/05/15 16:51, faraz khan wrote:
> Matt,
> Thanks for taking the time to go through this. Unfortunately this error
> only happens when WebRTC is acting as the server in the dtls handshake
> and no logs (at all) seem to be printed but I think that's because
> openssl logging is done in a different way in native WebRTC which I'm
> not handling. I'll try increasing logs and / or printing openssl errors.
> 
> The janus server gets the alert packet which I'm guessing is an error
> produced by the server in the transaction above.

Yes. You can see the following exchanges in the traces:

Client                    Server
------                    ------

ClientHello ------------>
                          ServerHello
                          Certificate
                          CertificateRequest
            <------------ ServerHelloDone
Certificate
ClientKeyExchange
CertificateVerify
ChangeCipherSpec
[Finished (Encrypted)] -->
            <------------ Alert (Decrypt Error)

If I understand your setup correctly (I'm not familiar with Janus or
WebRTC), Janus is acting as the client here and WebRTC is the server,
and both are using OpenSSL?

> 
> I'm assuming the problem is with the client reply that janus sent. Any
> idea what it could have sent wrong that WebRTC didn't like? Could it
> have encrypted the message with a wrong crypto or something or is this
> negotiation wholly handled by openssl? I'm trying to understand if janus
> is doing something wrong or is the dtls stuff completely handled by
> libssl - in which case this might be a dtls bug?

The server doesn't like something the Client sent in the last flight of
messages, i.e. the Certificate, ClientKeyExchange, CertificateVerify,
ChangeCipherSpec or Finished. It's unlikely to be the Certificate as, as
far as I can tell, its the same Certificate in the "good" exchange
versus the bad. So that means either the ClientKeyExchange,
CertificateVerify or Finished messages (CCS is super simple so I don't
think its that).

Is WebRTC using 1.0.1 too (and if so which letter version)? Looking at
the 1.0.1 code there are only a few places where a decrypt error can
come from. Possible causes (in 1.0.1 code base):

- digest check failed processing the Finished message
- CRL signature failure processing the client Certificate
- Bad RSA decrypt processing the CertificateVerify
- Bad RSA signature processing the CertificateVerify

There are a few other places that emit Decrypt Error alerts but none of
them seem to apply here. Getting some additional logging might help us
figure out which one of these is the cause.

These are all quite low level issues though...deep in OpenSSL internals.

Matt