[openssl-users] decrypt error

[Matt Caswell]

On 24/01/2019 09:19, Scharfenberg, Carsten wrote:
> Hello everybody,
>  
> I’ve just joined this group because I hope you guys can help me with my problem.
>  
> I'm using haproxy 1.8.17 and openssl 1.1.1a from Debian testing to serve TLS 1.2
> connections with client authentication.
> The TLS handshake runs through fine. But then the server answers with a Decrypt
> Error Alert to the Finish message sent by the client.

What is the client? Is this also OpenSSL (and if so which version), or is it
something else?

> How would I debug this error?
> Is it possible that something is wrong with my certificates?

That seems unlikely. If there was something wrong with the certificates I would
have expected the handshake to fail before the point that it gets to.

>  
> I’ve had a look into the source code. Unfortunately it’s not so easy to read. It
> seems to me the alert is generated here:
> ssl\statem\statem_lib.c:809 in function ‘tls_process_finished’ when the
> comparison of ‘pkt’ and ‘s->s3->tmp.peer_finish_md’ fails.
> Unfortunately I currently do not know what this means.

As each endpoint sends and receives handshake messages they both digest the
contents of those messages. The last step in the process is for each endpoint to
compare the digests that they created. They should be the same. If they are
different then that indicates that something has changed between one endpoint
sending one of those messages and its peer receiving it. This could in theory be
an active attacker. More likely though it could be due to some form of
corruption taking place somewhere. Some ideas as to the source of a possible
corruption:

- client application bug
- server application bug
- client TLS library bug
- server TLS library bug
- network "middlebox" corruption

I'd start by trying to isolate whether the problem is on the client side, the
server side, or the network. e.g. if the client is on the same host as the
server does the issue occur? Can you connect from a different client (different
application and/or different library or library version). Can the client connect
to other servers successfully.

Matt