DHE key exchange fails with the FIPS provider

[Thomas Dwyer III]

I'm having a problem connecting to particular machines via TLSv1.2 with 
the FIPS provider. The handshake fails with:

1022FDB6:error:0A000066:SSL routines:(unknown function):bad dh 
value:ssl/statem/statem_clnt.c:2085:

and I can't figure out what the problem is. The weird thing is the 
connection always succeeds with the default provider, but with the FIPS 
provider it works with some servers (e.g. oracle.com) and fails with 
other servers (e.g. debian.com). I have been able to reproduce the 
problem with the openssl command line using options that force the same 
cipher & key exchange that is negotiated by my application code:

openssl s_client -4 -tls1_2 -sigalgs rsa_pkcs1_sha256 -cipher 
DHE-RSA-AES128-GCM-SHA256 -trace -connect hostname:443

When using the FIPS provider and connecting to oracle.com this works 
fine. The exact same command line fails with debian.com. Connections to 
both machines work fine with the default provider. Both machines use 4K 
RSA certificates.

What is causing OpenSSL FIPS to fail the DHE key exchange?

Incidentally, setting "security-checks = 0" in the configuration file 
has no obvious effect on the problem.


Thanks,
Tom.III
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mta.openssl.org/pipermail/openssl-users/attachments/20230814/0ecf1ff8/attachment-0005.htm>