[openssl/openssl] a830f5: Limit the execution time of RSA public key check

[Tomáš Mráz]

  Branch: refs/heads/openssl-3.1
  Home:   https://github.com/openssl/openssl
  Commit: a830f551557d3d66a84bbb18a5b889c640c36294
      https://github.com/openssl/openssl/commit/a830f551557d3d66a84bbb18a5b889c640c36294
  Author: Tomas Mraz <tomas at openssl.org>
  Date:   2024-01-15 (Mon, 15 Jan 2024)

  Changed paths:
    M crypto/rsa/rsa_sp800_56b_check.c
    M test/recipes/91-test_pkey_check.t
    A test/recipes/91-test_pkey_check_data/rsapub_17k.pem

  Log Message:
  -----------
  Limit the execution time of RSA public key check

Fixes CVE-2023-6237

If a large and incorrect RSA public key is checked with
EVP_PKEY_public_check() the computation could take very long time
due to no limit being applied to the RSA public key size and
unnecessarily high number of Miller-Rabin algorithm rounds
used for non-primality check of the modulus.

Now the keys larger than 16384 bits (OPENSSL_RSA_MAX_MODULUS_BITS)
will fail the check with RSA_R_MODULUS_TOO_LARGE error reason.
Also the number of Miller-Rabin rounds was set to 5.

Reviewed-by: Neil Horman <nhorman at openssl.org>
Reviewed-by: Matt Caswell <matt at openssl.org>
(Merged from https://github.com/openssl/openssl/pull/23243)

(cherry picked from commit e09fc1d746a4fd15bb5c3d7bbbab950aadd005db)


  Commit: 3a665e45b8b08957d1ba9228bf0c9c31cff074e5
      https://github.com/openssl/openssl/commit/3a665e45b8b08957d1ba9228bf0c9c31cff074e5
  Author: Tomas Mraz <tomas at openssl.org>
  Date:   2024-01-15 (Mon, 15 Jan 2024)

  Changed paths:
    M CHANGES.md
    M NEWS.md

  Log Message:
  -----------
  Add CHANGES.md and NEWS.md entries for CVE-2023-6237

Reviewed-by: Neil Horman <nhorman at openssl.org>
Reviewed-by: Matt Caswell <matt at openssl.org>
(Merged from https://github.com/openssl/openssl/pull/23243)

(cherry picked from commit 38b2508f638787842750aec9a75745e1d8786743)


Compare: https://github.com/openssl/openssl/compare/60dc128b0f4a...3a665e45b8b0