[openssl/openssl] fc9867: Fix DH_check() excessive time with over sized modulus

[Matt Caswell]

  Branch: refs/heads/openssl-3.1
  Home:   https://github.com/openssl/openssl
  Commit: fc9867c1e03c22ebf56943be205202e576aabf23
      https://github.com/openssl/openssl/commit/fc9867c1e03c22ebf56943be205202e576aabf23
  Author: Matt Caswell <matt at openssl.org>
  Date:   2023-07-19 (Wed, 19 Jul 2023)

  Changed paths:
    M crypto/dh/dh_check.c
    M include/openssl/dh.h

  Log Message:
  -----------
  Fix DH_check() excessive time with over sized modulus

The DH_check() function checks numerous aspects of the key or parameters
that have been supplied. Some of those checks use the supplied modulus
value even if it is excessively large.

There is already a maximum DH modulus size (10,000 bits) over which
OpenSSL will not generate or derive keys. DH_check() will however still
perform various tests for validity on such a large modulus. We introduce a
new maximum (32,768) over which DH_check() will just fail.

An application that calls DH_check() and supplies a key or parameters
obtained from an untrusted source could be vulnerable to a Denial of
Service attack.

The function DH_check() is itself called by a number of other OpenSSL
functions. An application calling any of those other functions may
similarly be affected. The other functions affected by this are
DH_check_ex() and EVP_PKEY_param_check().

CVE-2023-3446

Reviewed-by: Paul Dale <pauli at openssl.org>
Reviewed-by: Tom Cosgrove <tom.cosgrove at arm.com>
Reviewed-by: Bernd Edlinger <bernd.edlinger at hotmail.de>
Reviewed-by: Tomas Mraz <tomas at openssl.org>
(Merged from https://github.com/openssl/openssl/pull/21451)

(cherry picked from commit 9e0094e2aa1b3428a12d5095132f133c078d3c3d)


  Commit: 4791e79b8803924b28c19af4d4036ad85335110d
      https://github.com/openssl/openssl/commit/4791e79b8803924b28c19af4d4036ad85335110d
  Author: Matt Caswell <matt at openssl.org>
  Date:   2023-07-19 (Wed, 19 Jul 2023)

  Changed paths:
    M test/dhtest.c

  Log Message:
  -----------
  Add a test for CVE-2023-3446

Confirm that the only errors DH_check() finds with DH parameters with an
excessively long modulus is that the modulus is too large. We should not
be performing time consuming checks using that modulus.

Reviewed-by: Paul Dale <pauli at openssl.org>
Reviewed-by: Tom Cosgrove <tom.cosgrove at arm.com>
Reviewed-by: Bernd Edlinger <bernd.edlinger at hotmail.de>
Reviewed-by: Tomas Mraz <tomas at openssl.org>
(Merged from https://github.com/openssl/openssl/pull/21451)

(cherry picked from commit ede782b4c8868d1f09c9cd237f82b6f35b7dba8b)


  Commit: 047a4c5c3284fc22f9842402ce210242d01df877
      https://github.com/openssl/openssl/commit/047a4c5c3284fc22f9842402ce210242d01df877
  Author: Matt Caswell <matt at openssl.org>
  Date:   2023-07-19 (Wed, 19 Jul 2023)

  Changed paths:
    M CHANGES.md
    M NEWS.md

  Log Message:
  -----------
  Update CHANGES/NEWS for CVE-2023-3446

Reviewed-by: Paul Dale <pauli at openssl.org>
Reviewed-by: Tom Cosgrove <tom.cosgrove at arm.com>
Reviewed-by: Bernd Edlinger <bernd.edlinger at hotmail.de>
Reviewed-by: Tomas Mraz <tomas at openssl.org>
(Merged from https://github.com/openssl/openssl/pull/21451)

(cherry picked from commit 4ec53ad6e1791daafbe26bdbd539f2ba9172959a)


Compare: https://github.com/openssl/openssl/compare/2c09df2807a1...047a4c5c3284