How to debug a TLSv1.3 protocol problem?

[Viktor Dukhovni]

On Mon, May 18, 2020 at 06:59:59PM +0200, Claus Assmann wrote:

> I'm stuck and looking for some hints/help.  I have two MTAs (let's
> call them M1 and S8), both built with OpenSSL 1.1.1g. The problem
> is M1 cannot establish a TLSv1.3 connection with S8. Using other
> MTAs/sites/protocols/tools works just fine, e.g., M1 can send mail
> to google using TLSv1.3, and S8 can send mail to M1. Replacing the
> server or client with openssl s_client/s_server also works.

I'll strongly second Matt's request for a PCAP file.

> M1 client side:
> apps_ssl_info_cb, where=10, ret=1
> apps_ssl_info_cb, SSL_connect=before SSL initialization
> ssl_msg_cb, writep=1, version=0, len=5, ct=100
> ssl_msg_cb, SSLv3/TLS write client hello
> ssl_msg_cb, writep=1, version=772, len=512, ct=16
> ssl_msg_cb, SSLv3/TLS write client hello
> apps_ssl_info_cb, SSL_connect=SSLv3/TLS write client hello
> ssl_msg_cb, writep=0, version=0, len=5, ct=100
> ssl_msg_cb, SSLv3/TLS write client hello
> apps_ssl_info_cb, SSL_connect=SSLv3/TLS write client hello
> ssl_msg_cb, writep=0, version=772, len=88, ct=16
> ssl_msg_cb, SSLv3/TLS read server hello
> apps_ssl_info_cb, SSL_connect=SSLv3/TLS read server hello
> ssl_msg_cb, writep=1, version=0, len=5, ct=100
> ssl_msg_cb, SSLv3/TLS write change cipher spec
> ssl_msg_cb, writep=1, version=772, len=1, ct=14
> ssl_msg_cb, SSLv3/TLS write change cipher spec
> apps_ssl_info_cb, SSL_connect=SSLv3/TLS write change cipher spec
> ssl_msg_cb, writep=1, version=0, len=5, ct=100
> ssl_msg_cb, SSLv3/TLS write client hello
> ssl_msg_cb, writep=1, version=772, len=512, ct=16
> ssl_msg_cb, SSLv3/TLS write client hello
> apps_ssl_info_cb, SSL_connect=SSLv3/TLS write client hello
> ssl_msg_cb, writep=0, version=0, len=5, ct=100
> ssl_msg_cb, SSLv3/TLS write client hello

The client trace looks rather odd, why is writing the hello
again after CCS?  I don't recall what happens with HRR
(Hello retry request) when client's initial keyshare is
not usable on the server...  Any unusual signature algorithm
preferences in this particular client?

-- 
    Viktor.