Issue in DH Algorithm Keys Generation in OpenSSL 3.3.0

[Vishal Kevat]

Hi Viktor,



I have assigned the task find out the root cause where the API is failing with this composite number. I see that with this composite number, the API BN_mod_inverse(Ri, R, &tmod, ctx) is returning NULL. (This is being called in bn_mont.c).

This function is defined in bn_gcd.c

Because of this API failed to return non-null value, the final API DH_generate_key() is failed to generate the DH public and private keys.



Can you explain what does the BN_mod_inverse() actually does.

Is this API related to the prime check on the DH Algorithm input prime number?



Regards,

Vishal






General

-----Original Message-----
From: openssl-users <openssl-users-bounces at openssl.org> On Behalf Of Viktor Dukhovni
Sent: Friday, May 31, 2024 06:14 PM
To: openssl-users at openssl.org
Subject: Re: Issue in DH Algorithm Keys Generation in OpenSSL 3.3.0



[External email: Use caution with links and attachments]



________________________________







On Fri, May 31, 2024 at 12:39:12PM +0000, Vishal Kevat via openssl-users wrote:



> Is there any way to make this prime number work by doing some

> modifications in the openssl source code.



It ISN'T a *prime* number.



> Like bypassing the OpenSSL DH prime check?



Why do you want to use a broken DH group?  Even if that 128-bit composite number were instead prime, it would still be way too small to offer any security.



It is hard to imagine how what you're asking for makes any sense.



--

    Viktor.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mta.openssl.org/pipermail/openssl-users/attachments/20240603/2f3e7da9/attachment.htm>