How to debug a TLSv1.3 protocol problem?

[Matt Caswell]

On 22/05/2020 13:09, Claus Assmann wrote:
> On Tue, May 19, 2020, Claus Assmann wrote:
> 
> Here's another (confusing) note: the (failing) S8 trace log:
> 
>> Sent Record
>> Header:
>>   Version = TLS 1.2 (0x303)
>>   Content Type = Handshake (22)
> ...
>>       extensions, length = 12
>>         extension_type=supported_versions(43), length=2
>>             TLS 1.3 (772)
>>         extension_type=key_share(51), length=2
>>             NamedGroup: secp256r1 (P-256) (23)
> 
> If I enable SSL_CTX_set_ecdh_auto() in S8 (-DLTS_EC=2)
> (instead of using EC_KEY_new_by_curve_name(NID_X9_62_prime256v1))
> then this changes to
> 	NamedGroup: ecdh_x25519 (29)
> and the handshake does not fail.

SSL_CTX_set_ecdh_auto() is a no-op in 1.1.1g:

#  define SSL_CTX_set_ecdh_auto(dummy, onoff)      ((onoff) != 0)

Its actually the default anyway - there is no need to call it.

I assume you were you previously calling SSL_CTX_set_tmp_ecdh()? That
would make more sense. Don't call SSL_CTX_set_tmp_ecdh(). That's a
legacy function and probably isn't what you want to do. It sets
precisely one supported ECDH group, rather than the list of supported
groups that you get normally (which includes both X25519, P-256 and some
others).


> That's somehow weird, because a different client uses secp256r1
> too and that handshake (with S8) does not fail.
> Of course it would be nice if the TLS handshake provides a
> better error message for this case.
> Since I cannot change all the S8 servers out there, it seems I have
> to figure out what is wrong(?) in M1 for this case.

TLSv1.3 clients select the (EC)DH group (aka curve) that they're going
to offer. Your M1 client is offering X25519:

        extension_type=key_share(51), length=38
            NamedGroup: ecdh_x25519 (29)
            key_exchange:  (len=32):
3E7E05E66F3F978082E7445E0A6FA9C73F4F4C1E6423AA3FAB7B80C8E521F629

Your S8 server (before the change you just made) has been configured to
not support X25519. Therefore the server forces a HelloRetryRequest -
effectively telling the client to retry with P-256 this time. It is
after this HRR that the client seems to go wrong.

Following the change you just made the server suddenly supports both
X25519 and P-256 and can accept the first ClientHello that the client
offers. Therefore there is no need to send the HRR, and it seems that
the client can complete the handshake successfully in that case. It
suggests to me that the bug is still there, its just being avoided due
to the lack of an HRR.

Matt