[openssl-users] BN_mod_inverse:no inverse when calling OCSP_basic_sign

[Eric To]

Version: openssl 1.0.1u
OS: RHEL 6.5
Arch: x86_64 GNU/Linux


I'm getting a critical error when openssl try to sign the basic response.
The overall basic response itself seems fine since openssl can still decode
the response if i use -no_signature_verify...

Without it, i get:
Response Verify Failure
140479349708456:error:04091077:rsa routines:INT_RSA_VERIFY:wrong signature
length:rsa_sign.c:186:
140479349708456:error:0D0C5006:asn1 encoding routines:ASN1_item_verify:EVP
lib:a_verify.c:218:
140479349708456:error:27069075:OCSP routines:OCSP_basic_verify:signature
failure:ocsp_vfy.c:105:


Any hint on troubleshooting this would be great. Here is how I call the
OCSP_basic_sign:

OCSP_BASICRESP *bresp;
X509 *signer;
EVP_PKEY *key;
...
OCSP_basic_sign(bresp,
                    signer, key,
                    EVP_sha1(),
                    NULL, 0);

But getting the following error internally from OCSP_basic_sign:

OPENSSL ERROR: error:0306E06C:bignum routines:BN_mod_inverse:no inverse
OPENSSL ERROR: error:0D0DC006:asn1 encoding routines:ASN1_item_sign_ctx:EVP
lib

Running it through dbg, it failed at ASN1_item_sign_ctx In
crypto/asn1/a_sign.c

303    if (!EVP_DigestSignUpdate(ctx, buf_in, inl)
304        || !EVP_DigestSignFinal(ctx, buf_out, &outl)) {
305        outl = 0;
306        ASN1err(ASN1_F_ASN1_ITEM_SIGN_CTX, ERR_R_EVP_LIB);
307        goto err;
308    }


(gdb) print inl
$1 = 284
(gdb) print outl
$2 = 256
(gdb) print *ctx
$4 = {digest = 0x7fb2ecf12b80, engine = 0x0, flags = 0, md_data =
0x7fb2ef1629d0, pctx = 0x7fb2ef162970,
  update = 0x7fb2ecc2515a <update>}
(gdb) print *ctx->digest
$5 = {type = 64, pkey_type = 65, md_size = 20, flags = 12, init =
0x7fb2ecc2513c <init>, update = 0x7fb2ecc2515a <update>,
  final = 0x7fb2ecc2518b <final>, copy = 0, cleanup = 0, sign =
0x7fb2ecbf0c70 <RSA_sign>, verify = 0x7fb2ecbf1568 <RSA_verify>,
  required_pkey_type = {6, 19, 0, 0, 0}, block_size = 64, ctx_size = 104,
md_ctrl = 0}
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mta.openssl.org/pipermail/openssl-users/attachments/20161005/b953b69d/attachment.html>