[openssl/openssl] e09fc1: Limit the execution time of RSA public key check

[Tomáš Mráz]

  Branch: refs/heads/master
  Home:   https://github.com/openssl/openssl
  Commit: e09fc1d746a4fd15bb5c3d7bbbab950aadd005db
      https://github.com/openssl/openssl/commit/e09fc1d746a4fd15bb5c3d7bbbab950aadd005db
  Author: Tomas Mraz <tomas at openssl.org>
  Date:   2024-01-15 (Mon, 15 Jan 2024)

  Changed paths:
    M crypto/rsa/rsa_sp800_56b_check.c
    M test/recipes/91-test_pkey_check.t
    A test/recipes/91-test_pkey_check_data/rsapub_17k.pem

  Log Message:
  -----------
  Limit the execution time of RSA public key check

Fixes CVE-2023-6237

If a large and incorrect RSA public key is checked with
EVP_PKEY_public_check() the computation could take very long time
due to no limit being applied to the RSA public key size and
unnecessarily high number of Miller-Rabin algorithm rounds
used for non-primality check of the modulus.

Now the keys larger than 16384 bits (OPENSSL_RSA_MAX_MODULUS_BITS)
will fail the check with RSA_R_MODULUS_TOO_LARGE error reason.
Also the number of Miller-Rabin rounds was set to 5.

Reviewed-by: Neil Horman <nhorman at openssl.org>
Reviewed-by: Matt Caswell <matt at openssl.org>
(Merged from https://github.com/openssl/openssl/pull/23243)


  Commit: 38b2508f638787842750aec9a75745e1d8786743
      https://github.com/openssl/openssl/commit/38b2508f638787842750aec9a75745e1d8786743
  Author: Tomas Mraz <tomas at openssl.org>
  Date:   2024-01-15 (Mon, 15 Jan 2024)

  Changed paths:
    M CHANGES.md
    M NEWS.md

  Log Message:
  -----------
  Add CHANGES.md and NEWS.md entries for CVE-2023-6237

Reviewed-by: Neil Horman <nhorman at openssl.org>
Reviewed-by: Matt Caswell <matt at openssl.org>
(Merged from https://github.com/openssl/openssl/pull/23243)


Compare: https://github.com/openssl/openssl/compare/486ab0fb003d...38b2508f6387