Handling signature_algorithm extension on TLS1.3 server

[Matt Caswell]

On 07/06/2019 07:27, Raja Ashok wrote:
> Thanks for the detailed explanation.
> 
> So rsaEncryption cert can do both RSASSA-PKCS-v1_5 and RSASSA-PSS type
> signature. And also the digital signature present on the cert can be of type
> RSASSA-PKCS-v1_5 or RSASSA-PSS.
> 
> Currently in 1.1.1c's has_usable_cert() function, digital signature (Issuer's
> signature) present on rsaEncryption cert type is not checked. So if TLS1.3
> client sends rsa_pss_rsae_xxx in "signature_algorithm" extension and if the
> server's rsaEncrypted cert has digital signature of type RSASSA-PKCS-v1_5, then
> it should not use that certificate but it is using currently.

There are two extensions to consider signature_algorithms and
signature_algorithms_cert. From RFC8446:

   TLS 1.3 provides two extensions for indicating which signature
   algorithms may be used in digital signatures.  The
   "signature_algorithms_cert" extension applies to signatures in
   certificates, and the "signature_algorithms" extension, which
   originally appeared in TLS 1.2, applies to signatures in
   CertificateVerify messages.

Looking at the code for has_usable_cert you can see it first checking to see if
it has a certificate that can sign in accordance with signature algorithms. And
then it goes on to check whether the signature in the certificate itself is
consistent with signature_algorithms_cert.

So, if signature_algorithms_cert does not contain rsa_pkcs1_* and the
certificate contains a PKCS1.5 signature, then it shouldn't be being used.

However the RFC then goes on to say:

   If no "signature_algorithms_cert" extension is
   present, then the "signature_algorithms" extension also applies to
   signatures appearing in certificates.

This was an area of some ambiguity in the TLSv1.2 spec where only
signature_algorithms exists. I believe it was common practice for
implementations to not check the signatures in certificates for conformance with
this (certainly that is the way OpenSSL behaves). The TLSv1.3 spec seems to be
more explicit about this. I would expect our TLSv1.2 implementation to continue
to operate as it did before so this additional checking of signatures in
certificates where only the signature_algorithms extensions is present should
only apply to TLSv1.3.

I don't see any code to do this in has_usable_cert so this looks like a
potential bug. Although possibly it was left out on purpose.

Ben Kaduk may have a view on this who implemented this code:

https://github.com/openssl/openssl/pull/5068/commits/e639c37bddea48334cb45d88d407c655641e1a35


Matt