TLS 1.2 handshake issue (Server Certificate request)

[Bashin, Vladimir]

Thank you very much, Michael - let us digest the information and present it to the customer.   I may probably come back with the follow up questions in case they say something worth passing to you...

Regards,
Vladimir Bashin 

-----Original Message-----
From: openssl-users <openssl-users-bounces at openssl.org> On Behalf Of Michael Wojcik
Sent: Friday, February 7, 2020 3:37 PM
To: openssl-users at openssl.org
Subject: RE: TLS 1.2 handshake issue (Server Certificate request) 

> From: openssl-users [mailto:openssl-users-bounces at openssl.org] On 
> Behalf Of Bashin, Vladimir
> Sent: Friday, February 07, 2020 11:25

> ... during that handshake the TLS server requests the client 
> Certificate but our TLS client responds with the Certificates Length 0 
> that causes the TLS server to respond with the Handshake Failure.

This means your client can't find a certificate to send. Usually this means that it doesn't have a certificate that:

- is among the ones it's been told, in some application-specific manner, it can use as a client certificate;
- it has the corresponding private key for;
- is valid (not expired, etc.);
- has the appropriate purpose (X.509v3 Extended Key Usage, if present, includes TLS Web Client Authententication), if the application makes this check;
- and meets the requirements listed in the server's CertificateRequest message

Note that last point. The CertificateRequest sent by the server specifies requirements and hints for the client certificate, as described below.

Also note that here "application" refers to both the end application and whatever TLS implementation it's using. Some TLS implementations do most or all of the certificate-selection work for the end application; others don't.

> ... that could be happening in case the client's certificate does not 
> match the server certificate

No, the client certificate doesn't have to match anything in the server certificate. What it has to satisfy are the CertificateRequest parameters.

See RFC 5246 (TLSv1.2) section 7.4.4. CertificateRequest contains three fields:

1. certificate_types, which is a list of certificate "types", by which the authors mean the kind of public key the certificate uses. The RFC specifies four possible values, but two of them are redundant (for historical reasons), so basically this comes down to RSA and/or DSA. RFC 8422 added ECDSA, and there are some less-common variants.

2. supported_signature_algorithms, which is a list of the signature-and-hash-algorithm pairs the server supports for client certificates. These are integer value pairs which are defined to represent algorithm pairs such as {sha1, rsa}. The RFC and its successors specify the standard values at the time each RFC is published, and IANA maintains the current list.

3. certificate_authorities, which is a list of distinguished names.

If the client sends a certificate, it MUST use a public key that matches one of the types in #1, and a signature algorithm that matches one of the pairs in #2.

The client MAY attempt to find a certificate signed by one of the authorities listed in #3. In practice, many client applications will honor this list, to the point of not sending a certificate at all if they don't have one signed by a CA on the server's list.

So you need to know what's in that CertificateRequest message. Since it's encrypted, you'll probably have to get either the server or the client to log it. Or maybe the client can be configured to log why it couldn't find a suitable client certificate.

Short version: TLS PKI is difficult. It's so difficult, and so security-critical, that I don't recommend people try to configure it without an expert. Unfortunately experts are hard to come by. (I'm certainly not one, despite having worked with TLS for a couple of decades now, and taken courses on it, and read books and other references...)

--
Michael Wojcik