How to Manually allocate BIGNUM ->d and set dmax, top values to create a Result Buffer in openssl 1.1.1 ?

[William Roberts]

On Tue, Dec 22, 2020 at 3:45 AM prudvi raj <rajprudvi98 at gmail.com> wrote:
>
> Hello all,
>
> We use a hardware accelerator to calculate BIGNUM rr = a^p mod m .( bn_mod_exp).  I am trying to rewrite that logic for openssl 1.1.1. Code snippet of custom bn_mod_exp function:
> --
>     if(rr->d)
>     {
>         OPENSSL_free(rr->d);
>     }
>     rr->d = ( BN_ULONG * )( malloc( m->top * sizeof(BN_ULONG) ) );
>     rr->top = m->top;
>     rr->dmax = m->top;
>     rr->neg = 0;
>
>     publicKeyData.operandALength = a->top * sizeof(BN_ULONG);
>     publicKeyData.operandA = ( System::BYTE * )( a->d );
>     publicKeyData.operandBLength = p->top * sizeof(BN_ULONG);
>     publicKeyData.operandB = ( System::BYTE * )( p->d );
>     publicKeyData.modulusLength = m->top * sizeof(BN_ULONG);
>     publicKeyData.modulus = ( System::BYTE * )( m->d );
>
>     publicKeyData.resultLength = m->top * sizeof(BN_ULONG);
>     publicKeyData.result = ( System::BYTE * )( rr->d );
>
>     calculate ( publicKeyData );    <<calculate fills out the Result Bytes in "rr->d" buffer.
> --
>  I found  a few 'get' functions (no set functions though) like -- bn_get_top , bn_get_dmax. These are in "bn_intern.c" , not in "bn_lib.c" (or BN API).
>    OPENSSL_free(rr->d)
>    rr->d = ( BN_ULONG * )( malloc( m->top * sizeof(BN_ULONG) ) );
>     rr->top = m->top;
>     rr->dmax = m->top;
>     rr->neg = 0
>
> As forward declarations are no longer allowed in openssl 1.1.1 , how to replicate above operations in openssl 1.1.1 ?
> Are there any Set functions for set, dmax , d values (allocate memory for rr->d) . ?!
> Please help me on this!!
>
> Thanks,
> Prudvi.
>

IIUC, this is just a side effect of not being able to access the RSA
structure directly like in openssl 1.0.2 days.
The function RSA_set0_key() will allow you to set D, and there are
routines for other portions of the struct as well.
When the structure went opaque, getter and setters we're added for
your use, see:
  - https://www.openssl.org/docs/man1.1.1/man3/RSA_set0_key.html

If you need to keep backwards compat with 1.0.2, you can define those
getter/setter functions when building with 1.0.2 in your source
code. However, it's strongly recommended to not be using 1.0.2.

Bill