DHE key exchange fails with the FIPS provider

[Tomas Mraz]

FIPS requires that the DH parameters used in case of protocols that
cannot transfer the q value (which TLS is) are known safe primes.
Apparently the debian.com server does not use safe prime DH parameters
with TLS-1.2. Nowadays it is better to just disable DHE ciphersuites
with FIPS for maximum interoperability.

Tomas Mraz, OpenSSL

On Mon, 2023-08-14 at 18:37 -0700, Thomas Dwyer III wrote:
>  I'm having a problem connecting to particular machines via TLSv1.2
> with the FIPS provider. The handshake fails with:
>  
>  1022FDB6:error:0A000066:SSL routines:(unknown function):bad dh
> value:ssl/statem/statem_clnt.c:2085:
>  
>  and I can't figure out what the problem is. The weird thing is the
> connection always succeeds with the default provider, but with the
> FIPS provider it works with some servers (e.g. oracle.com) and fails
> with other servers (e.g. debian.com). I have been able to reproduce
> the problem with the openssl command line using options that force
> the same cipher & key exchange that is negotiated by my application
> code:
>  
>  openssl s_client -4 -tls1_2 -sigalgs rsa_pkcs1_sha256 -cipher DHE-
> RSA-AES128-GCM-SHA256 -trace -connect hostname:443
>  
>  When using the FIPS provider and connecting to oracle.com this works
> fine. The exact same command line fails with debian.com. Connections
> to both machines work fine with the default provider. Both machines
> use 4K RSA certificates.
>  
>  What is causing OpenSSL FIPS to fail the DHE key exchange?
>  
>  Incidentally, setting "security-checks = 0" in the configuration
> file has no obvious effect on the problem.
>  
>  
>  Thanks,
>  Tom.III
>  
>  

-- 
Tomáš Mráz, OpenSSL