[openssl/openssl] 06bbee: Make DH_check set some error bits in recently adde...

[Bernd Edlinger]

  Branch: refs/heads/openssl-3.1
  Home:   https://github.com/openssl/openssl
  Commit: 06bbee97aaa6b1998d05e03825a750f40fc836ea
      https://github.com/openssl/openssl/commit/06bbee97aaa6b1998d05e03825a750f40fc836ea
  Author: Bernd Edlinger <bernd.edlinger at hotmail.de>
  Date:   2023-07-26 (Wed, 26 Jul 2023)

  Changed paths:
    M crypto/dh/dh_check.c

  Log Message:
  -----------
  Make DH_check set some error bits in recently added error

The pre-existing error cases where DH_check returned zero
are not related to the dh params in any way, but are only
triggered by out-of-memory errors, therefore having *ret
set to zero feels right, but since the new error case is
triggered by too large p values that is something different.
On the other hand some callers of this function might not
be prepared to handle the return value correctly but only
rely on *ret. Therefore we set some error bits in *ret as
additional safety measure.

Reviewed-by: Paul Dale <pauli at openssl.org>
Reviewed-by: Tomas Mraz <tomas at openssl.org>
(Merged from https://github.com/openssl/openssl/pull/21524)

(cherry picked from commit 81d10e61a4b7d5394d08a718bf7d6bae20e818fc)