How to debug a TLSv1.3 protocol problem?

[Alexander Gryanko]

Hi,

You can try to add SSL_CTX_set_keylog_callback to your MTA. With keylog
callback, your MTA will start writing log in NSS format for decryption of
TLS frames. https://wiki.wireshark.org/TLS#Using_the_.28Pre.29-Master-Secret

Or you can enable SSL_trace with disabling OPENSSL_NO_SSL_TRACE build
option and  SSL_set_msg_callback(client_ssl, SSL_trace) callback.

But first of all, check your cert type. Looks like you are using non-RSA
cert which is not supported by S8.

On Mon, 18 May 2020 at 20:00, Claus Assmann <ca+ssl-users at esmtp.org> wrote:

> I'm stuck and looking for some hints/help.  I have two MTAs (let's
> call them M1 and S8), both built with OpenSSL 1.1.1g. The problem
> is M1 cannot establish a TLSv1.3 connection with S8. Using other
> MTAs/sites/protocols/tools works just fine, e.g., M1 can send mail
> to google using TLSv1.3, and S8 can send mail to M1. Replacing the
> server or client with openssl s_client/s_server also works.
>
> I've added some TLS callbacks to S8 which I found in s_cb.c, but
> all I get at the end is "SSL_accept:error in TLSv1.3 early data"
> (see "full" trace below for the steps leading to this).
> Unfortunately I cannot find a way to figure out more details or
> what kind of error that is.  Any hints how to determine (and fix?)
> the problem?
>
> S8 server side:
> info_callback where=0x10, ret=1
> info_callback where=0x2001, ret=1
> SSL_accept:before SSL initialization
> ssl_msg_cb, writep=0, version=0, len=5, ct=256
> ssl_msg_cb, before SSL initialization
> info_callback where=0x2001, ret=1
> SSL_accept:before SSL initialization
> ssl_msg_cb, writep=0, version=772, len=512, ct=22
> ssl_msg_cb, SSLv3/TLS read client hello
> info_callback where=0x2001, ret=1
> SSL_accept:SSLv3/TLS read client hello
> ssl_msg_cb, writep=1, version=0, len=5, ct=256
> ssl_msg_cb, SSLv3/TLS write server hello
> ssl_msg_cb, writep=1, version=772, len=88, ct=22
> ssl_msg_cb, SSLv3/TLS write server hello
> info_callback where=0x2001, ret=1
> SSL_accept:SSLv3/TLS write server hello
> ssl_msg_cb, writep=1, version=0, len=5, ct=256
> ssl_msg_cb, SSLv3/TLS write change cipher spec
> ssl_msg_cb, writep=1, version=772, len=1, ct=20
> ssl_msg_cb, SSLv3/TLS write change cipher spec
> info_callback where=0x2001, ret=1
> SSL_accept:SSLv3/TLS write change cipher spec
> info_callback where=0x2001, ret=1
> SSL_accept:TLSv1.3 early data
> info_callback where=0x2002, ret=-1
> SSL_accept:error in TLSv1.3 early data
>
> M1 client side:
> apps_ssl_info_cb, where=10, ret=1
> apps_ssl_info_cb, SSL_connect=before SSL initialization
> ssl_msg_cb, writep=1, version=0, len=5, ct=100
> ssl_msg_cb, SSLv3/TLS write client hello
> ssl_msg_cb, writep=1, version=772, len=512, ct=16
> ssl_msg_cb, SSLv3/TLS write client hello
> apps_ssl_info_cb, SSL_connect=SSLv3/TLS write client hello
> ssl_msg_cb, writep=0, version=0, len=5, ct=100
> ssl_msg_cb, SSLv3/TLS write client hello
> apps_ssl_info_cb, SSL_connect=SSLv3/TLS write client hello
> ssl_msg_cb, writep=0, version=772, len=88, ct=16
> ssl_msg_cb, SSLv3/TLS read server hello
> apps_ssl_info_cb, SSL_connect=SSLv3/TLS read server hello
> ssl_msg_cb, writep=1, version=0, len=5, ct=100
> ssl_msg_cb, SSLv3/TLS write change cipher spec
> ssl_msg_cb, writep=1, version=772, len=1, ct=14
> ssl_msg_cb, SSLv3/TLS write change cipher spec
> apps_ssl_info_cb, SSL_connect=SSLv3/TLS write change cipher spec
> ssl_msg_cb, writep=1, version=0, len=5, ct=100
> ssl_msg_cb, SSLv3/TLS write client hello
> ssl_msg_cb, writep=1, version=772, len=512, ct=16
> ssl_msg_cb, SSLv3/TLS write client hello
> apps_ssl_info_cb, SSL_connect=SSLv3/TLS write client hello
> ssl_msg_cb, writep=0, version=0, len=5, ct=100
> ssl_msg_cb, SSLv3/TLS write client hello
>
> and here it hangs until timeout.
>
>

-- 
С уважением,
Александр Грянко
Phone: +7(962)9558222
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mta.openssl.org/pipermail/openssl-users/attachments/20200518/6eb437c4/attachment.html>