[openssl/openssl] 0b0f7a: Limit the execution time of RSA public key check

[Tomáš Mráz]

  Branch: refs/heads/openssl-3.2
  Home:   https://github.com/openssl/openssl
  Commit: 0b0f7abfb37350794a4b8960fafc292cd5d1b84d
      https://github.com/openssl/openssl/commit/0b0f7abfb37350794a4b8960fafc292cd5d1b84d
  Author: Tomas Mraz <tomas at openssl.org>
  Date:   2024-01-15 (Mon, 15 Jan 2024)

  Changed paths:
    M crypto/rsa/rsa_sp800_56b_check.c
    M test/recipes/91-test_pkey_check.t
    A test/recipes/91-test_pkey_check_data/rsapub_17k.pem

  Log Message:
  -----------
  Limit the execution time of RSA public key check

Fixes CVE-2023-6237

If a large and incorrect RSA public key is checked with
EVP_PKEY_public_check() the computation could take very long time
due to no limit being applied to the RSA public key size and
unnecessarily high number of Miller-Rabin algorithm rounds
used for non-primality check of the modulus.

Now the keys larger than 16384 bits (OPENSSL_RSA_MAX_MODULUS_BITS)
will fail the check with RSA_R_MODULUS_TOO_LARGE error reason.
Also the number of Miller-Rabin rounds was set to 5.

Reviewed-by: Neil Horman <nhorman at openssl.org>
Reviewed-by: Matt Caswell <matt at openssl.org>
(Merged from https://github.com/openssl/openssl/pull/23243)

(cherry picked from commit e09fc1d746a4fd15bb5c3d7bbbab950aadd005db)


  Commit: 20cb674d81e8070439d50c41f340ed439435aa84
      https://github.com/openssl/openssl/commit/20cb674d81e8070439d50c41f340ed439435aa84
  Author: Tomas Mraz <tomas at openssl.org>
  Date:   2024-01-15 (Mon, 15 Jan 2024)

  Changed paths:
    M CHANGES.md
    M NEWS.md

  Log Message:
  -----------
  Add CHANGES.md and NEWS.md entries for CVE-2023-6237

Reviewed-by: Neil Horman <nhorman at openssl.org>
Reviewed-by: Matt Caswell <matt at openssl.org>
(Merged from https://github.com/openssl/openssl/pull/23243)

(cherry picked from commit 38b2508f638787842750aec9a75745e1d8786743)


Compare: https://github.com/openssl/openssl/compare/9aee66b2f4ec...20cb674d81e8