[openssl/openssl] 9ab1c7: Fix error propagatation in BN_check_prime()

[Kazuki Yamaguchi]

  Branch: refs/heads/openssl-3.0
  Home:   https://github.com/openssl/openssl
  Commit: 9ab1c7612ccadba86949c5ebbf4ecc9f32432af7
      https://github.com/openssl/openssl/commit/9ab1c7612ccadba86949c5ebbf4ecc9f32432af7
  Author: Kazuki Yamaguchi <k at rhe.jp>
  Date:   2022-10-04 (Tue, 04 Oct 2022)

  Changed paths:
    M crypto/bn/bn_prime.c

  Log Message:
  -----------
  Fix error propagatation in BN_check_prime()

BN_check_prime() is supposed to return 0 for a composite number and -1
on error. Properly translate the return value of the internal function
ossl_bn_miller_rabin_is_prime(), where 0 means an error.

The confusion prevented BN_GENCB callbacks from aborting the primality
test or key generation routines utilizing this.

Reviewed-by: Tomas Mraz <tomas at openssl.org>
Reviewed-by: Nicola Tuveri <nic.tuv at gmail.com>
Reviewed-by: Paul Dale <pauli at openssl.org>
(Merged from https://github.com/openssl/openssl/pull/19314)

(cherry picked from commit 0b3867634f74f6cb7b60b3a0adde396421207214)