[openssl/openssl] 18c024: Limit the execution time of RSA public key check

[Tomáš Mráz]

  Branch: refs/heads/openssl-3.0
  Home:   https://github.com/openssl/openssl
  Commit: 18c02492138d1eb8b6548cb26e7b625fb2414a2a
      https://github.com/openssl/openssl/commit/18c02492138d1eb8b6548cb26e7b625fb2414a2a
  Author: Tomas Mraz <tomas at openssl.org>
  Date:   2024-01-15 (Mon, 15 Jan 2024)

  Changed paths:
    M crypto/rsa/rsa_sp800_56b_check.c
    M test/recipes/91-test_pkey_check.t
    A test/recipes/91-test_pkey_check_data/rsapub_17k.pem

  Log Message:
  -----------
  Limit the execution time of RSA public key check

Fixes CVE-2023-6237

If a large and incorrect RSA public key is checked with
EVP_PKEY_public_check() the computation could take very long time
due to no limit being applied to the RSA public key size and
unnecessarily high number of Miller-Rabin algorithm rounds
used for non-primality check of the modulus.

Now the keys larger than 16384 bits (OPENSSL_RSA_MAX_MODULUS_BITS)
will fail the check with RSA_R_MODULUS_TOO_LARGE error reason.
Also the number of Miller-Rabin rounds was set to 5.

Reviewed-by: Neil Horman <nhorman at openssl.org>
Reviewed-by: Matt Caswell <matt at openssl.org>
(Merged from https://github.com/openssl/openssl/pull/23243)

(cherry picked from commit e09fc1d746a4fd15bb5c3d7bbbab950aadd005db)


  Commit: 5149deb832bc98cd908c4fca345c6f1150f4e286
      https://github.com/openssl/openssl/commit/5149deb832bc98cd908c4fca345c6f1150f4e286
  Author: Tomas Mraz <tomas at openssl.org>
  Date:   2024-01-15 (Mon, 15 Jan 2024)

  Changed paths:
    M CHANGES.md
    M NEWS.md

  Log Message:
  -----------
  Add CHANGES.md and NEWS.md entries for CVE-2023-6237

Reviewed-by: Neil Horman <nhorman at openssl.org>
Reviewed-by: Matt Caswell <matt at openssl.org>
(Merged from https://github.com/openssl/openssl/pull/23243)

(cherry picked from commit 38b2508f638787842750aec9a75745e1d8786743)


Compare: https://github.com/openssl/openssl/compare/11f7b60b6ff4...5149deb832bc