[openssl-1.1.1l] TLS1.2 Server responses with Alert

[Mark Hack]

The server error is correct - the signature_algorithms_cert extension
does not offer rsa_pkcs1_sha256 (0x0401) which is the server
certificate signing algorithm.
If the client is written in Java, check java.security  for
"jdk.certpath.disabledAlgorithms"  and check the constraints.


On Fri, 2021-12-31 at 15:05 +0000, Michael Wojcik wrote:
> > From: openssl-users <openssl-users-bounces at openssl.org> On Behalf
> > Of Ma Zhenhua
> > Sent: Thursday, 30 December, 2021 23:59
> > On the SSL/TLS server, there's one error as follows. 
> > "SSL Error(118) - no suitable signature algorithm"
> 
> Debugging handshake failures isn't my area of expertise, but I note
> both ClientHellos include a signature_algorithms extension, and the
> contents are quite different. In particular, the successful
> ClientHello includes the Signature Hash Algorithm Hash and Signature
> Hash Algorithm Signature parameters, while the failing one doesn't.
> 
> The failing one also includes a signature_algorithms_cert extension,
> while the successful one does not. I don't know offhand how the
> algorithms specified in that extension correspond to the signature-
> algorithm OIDs in signatures, but the server's certificate has
> 1.2.840.113549.1.1.11 (sha256WithRSAEncryption) which seems like it
> ought to correspond to either rsa_pss_rsae_sha256 or
> rsa_pss_pss_sha256. (Apparently those are both RSA-PSS with SHA256,
> as the name implies, and the difference between the two of them is
> whether the public key is encoded using the rsaEncryption format in
> the certificate, or the id-RSASSA-PSS format. The failing client is
> saying it understands both, AIUI.)
> 
> So my guess would be the server is unhappy that the failing client's
> ClientHello doesn't include the parameters for the various supported
> signature schemes in its signature_algorithms extension. But that's
> just a guess, and I don't know how you'd fix it.
> 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mta.openssl.org/pipermail/openssl-users/attachments/20220101/8dfee888/attachment-0001.htm>