OpenSSL 3.0 different behaviour on smaller DH groups?

[Simon Chopin]

Hi,

I'm working on migrating the Ruby net-ssh package to OpenSSL 3.0 as part
of our larger transition in Ubuntu, but there's an issue that I can't
figure out.

This test suite fails several times with a failed call to
EVP_PKEY_derive_set_peer, without much more details:
https://github.com/net-ssh/net-ssh/blob/master/test/transport/kex/test_diffie_hellman_group14_sha1.rb

However, the *exact same* test suite works, with the only difference
being that the failing suite uses the DH group 14, which is 2048bits,
whereas the one that passes uses the group 1, which the Internet tells
me is 768bits.

I'm working on trying to come up with a pure-C reproducer, but I'm
sending this in the hopes that someone out there has a sudden epiphany
and knows what is causing this?

For the curious out there, we're tracking this work at
https://bugs.launchpad.net/ubuntu/+source/vagrant/+bug/1964025

Cheers,

--
Simon Chopin
Foundations Team                               	             Ubuntu MOTU
simon.chopin at canonical.com                            schopin at ubuntu.com