[openssl-1.1.1l] TLS1.2 Server responses with Alert

[Michael Wojcik]

> From: openssl-users <openssl-users-bounces at openssl.org> On Behalf Of Ma Zhenhua
> Sent: Thursday, 30 December, 2021 23:59

> On the SSL/TLS server, there's one error as follows. 
> "SSL Error(118) - no suitable signature algorithm"

Debugging handshake failures isn't my area of expertise, but I note both ClientHellos include a signature_algorithms extension, and the contents are quite different. In particular, the successful ClientHello includes the Signature Hash Algorithm Hash and Signature Hash Algorithm Signature parameters, while the failing one doesn't.

The failing one also includes a signature_algorithms_cert extension, while the successful one does not. I don't know offhand how the algorithms specified in that extension correspond to the signature-algorithm OIDs in signatures, but the server's certificate has 1.2.840.113549.1.1.11 (sha256WithRSAEncryption) which seems like it ought to correspond to either rsa_pss_rsae_sha256 or rsa_pss_pss_sha256. (Apparently those are both RSA-PSS with SHA256, as the name implies, and the difference between the two of them is whether the public key is encoded using the rsaEncryption format in the certificate, or the id-RSASSA-PSS format. The failing client is saying it understands both, AIUI.)

So my guess would be the server is unhappy that the failing client's ClientHello doesn't include the parameters for the various supported signature schemes in its signature_algorithms extension. But that's just a guess, and I don't know how you'd fix it.

-- 
Michael Wojcik