Handling signature_algorithm extension on TLS1.3 server

[Matt Caswell]

On 06/06/2019 16:15, Raja Ashok wrote:
> Hi,
> 
> Currently has_usable_cert() function is called on tls_choose_sigalg() to find
> out the suitable certificate available. But currently rsa_pkcs1_xxx and
> rsa_pss_rsae_xxx certs are stored on same index SSL_PKEY_RSA. Because of this it
> may ends in choosing rsa_pkcs1_xxx cert for rsa_pss_rsae_xxx extension. Is this
> behaviour correct ?

There are two things to consider:

1) The OID in the RSA cert, which can be one of rsaEncryption or RSASSA-PSS. The
former is for "traditional" RSA certs, whilst the latter *only* allows use of
the key for signing (it cannot be used for encryption).

2) The type of signing in use, e.g. RSASSA-PKCS-v1_5 or RSASSA-PSS.

rsaEncryption certs are capable of doing *either* form of signing, whilst
RSASSA-PSS certs can only do PSS signing.

We store rsaEncryption certs under the SSL_PKEY_RSA index, and RSASSA-PSS certs
under the SSL_PKEY_RSA_PSS_SIGN index.

TLSv1.2 and below signs handshake messages using PKCS v1.5. which corresponds to
these signature algorithms:

          rsa_pkcs1_sha256(0x0401)
          rsa_pkcs1_sha384(0x0501)
          rsa_pkcs1_sha512(0x0601)

These sig algs cannot be used in TLSv1.3 for signing handshake messages,
although they may appear in a ClientHello for backwards compatibility with
TLSv1.2. You can only use these sig algs with "traditional" RSA certs (not PSS
RSA certs).

TLSv1.3 signs handshake messages using PSS which corresponds to these signature
algorithms for "traditional" (rsaEncryption) certs:

          rsa_pss_rsae_sha256(0x0804)
          rsa_pss_rsae_sha384(0x0805)
          rsa_pss_rsae_sha512(0x0806)

Or these signature algorithms for PSS certs:

          rsa_pss_pss_sha256(0x0809)
          rsa_pss_pss_sha384(0x080a)
          rsa_pss_pss_sha512(0x080b)

Therefore it is perfectly correct behaviour that a cert stored under the
SSL_PKEY_RSA index could be used for signing handshake message using either
rsa_pkcs1_xxx or for rsa_pss_rsae_xxx. The former is used in TLSv1.2 and the
latter is used in TLSv1.3.

Matt