no suitable signature algorithm during handshake failure

[Quanah Gibson-Mount]

Working on a migration for an application (OpenLDAP) where the old version 
is linked to OpenSSL 1.0.2 to where the new version is linked to OpenSSL 
1.1.1h.

Most client applications are working without issue.  However, one Windows 
client application consistently fails to connect to the OpenSSL 1.1.1h 
linked slapd with an error of no suitable signature algorithm during the 
handshake.

Using wireshark, we can see the following signature algorithms are offered 
from the client side (which uses TLSv1.2) for both the working and failing 
servers:

0x0403 ECDSA-SHA256
0x0503 ECDSA-SHA384
0x0603 ECDSA-SHA512
0x0401 RSA-SHA256
0x0501 RSA-SHA384
0x0601 RSA-SHA512
0x0402 DSA-SHA256
0x0203 ECDSA-SHA1
0x0201 RSA-SHA1
0x0202 DSA-SHA1

If I test connecting on the command line to the server in question, I can 
connect using any of RSA+SHA256, RSA+SHA384, and RSA+SHA512 from the above 
signature algorithms without issue, like:

openssl s_client -connect <host:636> -tls1_2 -sigalgs RSA+SHA256

Any suggestions as to why the windows client is unable to negotiate with a 
new version of OpenSSL?

The error in the log is:

error: 14201076:SSL routines:tls_choose_sigalg:no suitable signature 
algorithm.

Thanks,
Quanah

--

Quanah Gibson-Mount
Product Architect
Symas Corporation
Packaged, certified, and supported LDAP solutions powered by OpenLDAP:
<http://www.symas.com>