[openssl-users] Fwd: issue with dtls failure during openssl upgrade from 1.0.1m to q

[Matt Caswell]

Hello

On 17/06/16 12:04, Test ssl wrote:
> We are facing an issue with DTLS failure during the Openssl upgrade from
> 1.0.1m to 1.0.1q. We have attached the network trace file in attachment
> with good (1.0.1m) and fail (1.0.1q) case.
> 
> The test scenario is that we are trying to connect a cisco based
> Endpoint device to a Video conference server, where our code resides.
> 
> Please help us with this DTLS failure scenario which we are not able to
> understand that what wrong we did in our application code in using
> Openssl APIs.

This looks like quite a strange trace in both the success and failure cases.

The traces are slightly confusing because it looks like you are making
multiple connections from different ports. To simplify things in the
success case I filtered wireshark to only show me the port 22602 <->
port 49164 communication.

This showed me the following interaction


Client                     Server
======                     ======

ClientHello
                           ServerHello
                           Certificate
                           ServerHelloDone
ClientKeyExchange
ChangeCipherSpec
EncryptedHandshakeMessage
                           ChangeCipherSpec
                           EncryptedHandshakeMessage

So far so good. This looks like a normal successful handshake. The
EncryptedHandshakeMessages at the end of these exchanges are the
Finished messages indicating that the handshake was successful. I would
then expect to see Application Data being exchanged. Instead I see this:

Client                     Server
======                     ======

                           ClientHello
ServerHello
Certificate
ServerKeyExchange
CertificateRequest
ServerHelloDone
                           Certificate
                           ClientKeyExchange
                           CertificateVerify
                           ChangeCipherSpec
                           EncryptedHandshakeMessage
ChangeCipherSpec
EncryptedHandshakeMessage

So AFAICT the client and server appear to have swapped roles!!! The
server is sending Client message and the client is sending server
messages. Not only that but the connection previously established seems
to have been thrown away and they are starting from scratch (i.e.
unencrypted) without having first shutdown the previous connection or
having sent any application data.

The failure case is similar. The client and server apparently
successfully complete a handshake. They then seem to swap roles
(starting from scratch again, without any app data being sent), except
this time we see:

Client                     Server
======                     ======

                           ClientHello

ServerHello
Certificate
ServerKeyExchange
CertificateRequest
ServerHelloDone
                           Alert (Handshake Failure)
                           ClientHello
ServerHello
Certificate
ServerKeyExchange
CertificateRequest
ServerHelloDone
                           Alert (Decrypt Error)


So here we see the "server", that seems to have forgotten it is the
server and is now acting as the client attempt to initiate a handshake
(forgetting all about the previous connection). It then fails with a
Handshake Failure (I'm not surprised), and apparently decides to have
another go. This time we see a DecryptError which suggests the "server"
(acting as a client) is expecting to receive encrypted data, but the
client (acting as a server) is still sending unencrypted data.

You need to try and figure out why the two ends of the communication are
so confused about what role they are playing (or is there something you
haven't told me about the way your application works?).

Matt