DHE key exchange fails with the FIPS provider
tomas at openssl.org
Tue Aug 15 06:47:29 UTC 2023
FIPS requires that the DH parameters used in case of protocols that
cannot transfer the q value (which TLS is) are known safe primes.
Apparently the debian.com server does not use safe prime DH parameters
with TLS-1.2. Nowadays it is better to just disable DHE ciphersuites
with FIPS for maximum interoperability.
Tomas Mraz, OpenSSL
On Mon, 2023-08-14 at 18:37 -0700, Thomas Dwyer III wrote:
> I'm having a problem connecting to particular machines via TLSv1.2
> with the FIPS provider. The handshake fails with:
> 1022FDB6:error:0A000066:SSL routines:(unknown function):bad dh
> and I can't figure out what the problem is. The weird thing is the
> connection always succeeds with the default provider, but with the
> FIPS provider it works with some servers (e.g. oracle.com) and fails
> with other servers (e.g. debian.com). I have been able to reproduce
> the problem with the openssl command line using options that force
> the same cipher & key exchange that is negotiated by my application
> openssl s_client -4 -tls1_2 -sigalgs rsa_pkcs1_sha256 -cipher DHE-
> RSA-AES128-GCM-SHA256 -trace -connect hostname:443
> When using the FIPS provider and connecting to oracle.com this works
> fine. The exact same command line fails with debian.com. Connections
> to both machines work fine with the default provider. Both machines
> use 4K RSA certificates.
> What is causing OpenSSL FIPS to fail the DHE key exchange?
> Incidentally, setting "security-checks = 0" in the configuration
> file has no obvious effect on the problem.
Tomáš Mráz, OpenSSL
More information about the openssl-users