[External] : Re: DHE key exchange fails with the FIPS provider

[Thomas Dwyer III]

Following up on this, I have legacy applications that are FIPS-enabled 
with OpenSSL 1.0.2+FOM that I'm porting to OpenSSL 3.x and these 
applications are failing because of the now-invalid DH parameters that 
weren't invalid in 1.0.2. I understand that compliance based on whether 
the parameters are validated using 186-4 vs 186-2 is a question for the 
FIPS lab rather than the OpenSSL community, but if I wanted to revert 
back to 186-2 validation in the FIPS provider and submit that to CMVP 
what would be the best way to achieve this? I see this comment in 
ffc_params_validate.c directly above the definition of 
ossl_ffc_params_FIPS186_2_validate():

     /* This may be used in FIPS mode to validate deprecated FIPS-186-2 
Params */
     int ossl_ffc_params_FIPS186_2_validate(OSSL_LIB_CTX *libctx,
                                            const FFC_PARAMS *params, 
int type,
                                            int *res, BN_GENCB *cb)

but I can't find a codepath through the FIPS provider that gets me 
there. All of the logic that checks FFC_PARAM_FLAG_VALIDATE_LEGACY is 
removed when FIPS_MODULE is defined. Is the comment incorrect?

A heavy-handed approach would be to #undef FIPS_MODULE for the relevant 
source files but that would also affect parameter generation, not just 
validation. I tried just simply changing all calls to 
ossl_ffc_params_FIPS186_4_validate() to instead call 
ossl_ffc_params_FIPS186_2_validate() but I must be missing something 
else because that didn't work.

By the way, I'm seeing surprising behavior on the command line. With 
configuration files that load & activate the base & fips providers 
(leaving the default provider inactive) and this command line:

     openssl dhparam 2048 | openssl3 dhparam -noout -check -provider 
default -propquery fips=yes

(where "openssl" is 1.1.1 and "openssl3" is 3.0.x) I see the following:

     DH parameters appear to be ok.

Changing that to "-provider fips" changed the output to:

     Error, invalid parameters generated

Why does the first command line succeed? Shouldn't -propquery have 
forced dhparam to use the FIPS provider in both cases? Furthermore, when 
omitting the -provider option and just specifying "-propquery fips=no" I 
would have expected this to fail because nothing in my configuration 
activates the default provider, but that's not what happens--it's using 
the FIPS provider anyway and failing with "invalid parameters 
generated". It appears that -propquery is being ignored here?

Other commands (e.g. dgst) don't appear to have this issue:

     $ openssl3 dgst -md5 -provider default -propquery fips=no < /etc/motd
     MD5(stdin)= 0a38e488efc2773c0f3f40438c84f0ed
     $ openssl3 dgst -md5 -provider default -propquery fips=yes < /etc/motd
     dgst: Unknown option or message digest: md5
     dgst: Use -help for summary.
     0051636DCC7F0000:error:0308010C:digital envelope 
routines:inner_evp_generic_fetch:unsupported:crypto/evp/evp_fetch.c:373:Global 
default library context, Algorithm (md5 : 102), Properties (fips=yes)


Thanks,
Tom.III


On 8/14/23 23:47, Tomas Mraz wrote:
> FIPS requires that the DH parameters used in case of protocols that
> cannot transfer the q value (which TLS is) are known safe primes.
> Apparently the debian.com server does not use safe prime DH parameters
> with TLS-1.2. Nowadays it is better to just disable DHE ciphersuites
> with FIPS for maximum interoperability.
>
> Tomas Mraz, OpenSSL
>
> On Mon, 2023-08-14 at 18:37 -0700, Thomas Dwyer III wrote:
>>   I'm having a problem connecting to particular machines via TLSv1.2
>> with the FIPS provider. The handshake fails with:
>>   
>>   1022FDB6:error:0A000066:SSL routines:(unknown function):bad dh
>> value:ssl/statem/statem_clnt.c:2085:
>>   
>>   and I can't figure out what the problem is. The weird thing is the
>> connection always succeeds with the default provider, but with the
>> FIPS provider it works with some servers (e.g. oracle.com) and fails
>> with other servers (e.g. debian.com). I have been able to reproduce
>> the problem with the openssl command line using options that force
>> the same cipher & key exchange that is negotiated by my application
>> code:
>>   
>>   openssl s_client -4 -tls1_2 -sigalgs rsa_pkcs1_sha256 -cipher DHE-
>> RSA-AES128-GCM-SHA256 -trace -connect hostname:443
>>   
>>   When using the FIPS provider and connecting to oracle.com this works
>> fine. The exact same command line fails with debian.com. Connections
>> to both machines work fine with the default provider. Both machines
>> use 4K RSA certificates.
>>   
>>   What is causing OpenSSL FIPS to fail the DHE key exchange?
>>   
>>   Incidentally, setting "security-checks = 0" in the configuration
>> file has no obvious effect on the problem.
>>   
>>   
>>   Thanks,
>>   Tom.III
>>   
>>   
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mta.openssl.org/pipermail/openssl-users/attachments/20230818/31d2c295/attachment-0002.htm>