RSA_generate_key_ex is crashing when compiled on RHEL6 PPC and executed on RHEL8 for OpenSSL 3.0.1PPC

[Tomas Mraz]

Can you please try to build the openssl with debug information (-d on
Configure command line)? To see whether the backtrace will contain more
information.

Tomas Mraz

On Thu, 2022-06-02 at 11:09 +0530, Minal Patil wrote:
> Hello All,
> 
> I am trying to use RSA_generate_key_ex function to generate the RSA
> key pairs on RHEL 7.2 PPCle. I am observing crash when i link the
> source code with Openssl 3.0 whereas same works if i link with
> Openssl 1.0.2
> 
> Below is configure command used for compiling openssl
> ./Configure no-shared threads --prefix=/home/testuser/OpenSSL/Build -
> -openssldir=/home/testuser/OpenSSL/Build --libdir=lib linux-ppc64le -
> Wa,--noexecstack
> 
> I am attaching the source code I am using for reference. 
> Below is stack trace observed 
> 
> Program received signal SIGILL, Illegal instruction.
> #0  0x0000000010000004 in ?? ()
> #1  0x000000001005afdc in ossl_pool_acquire_entropy ()
> #2  0x00000000101fed14 in seed_get_seed ()
> #3  0x00000000102be844 in get_entropy ()
> #4  0x00000000102bee8c in ossl_prov_drbg_instantiate ()
> #5  0x00000000101f9b2c in drbg_ctr_instantiate_wrapper ()
> #6  0x00000000100ff2e0 in EVP_RAND_instantiate ()
> #7  0x0000000010031c18 in rand_new_drbg ()
> #8  0x0000000010032bd0 in RAND_get0_primary ()
> #9  0x00000000100333f8 in RAND_get0_private ()
> #10 0x0000000010033558 in RAND_priv_bytes_ex ()
> #11 0x00000000100926cc in BN_priv_rand_ex ()
> #12 0x0000000010090b78 in BN_generate_prime_ex2 ()
> #13 0x0000000010035a90 in RSA_generate_multi_prime_key ()
> #14 0x00000000100361b4 in RSA_generate_key_ex ()
> #15 0x00000000100048b8 in generate_key (keysize=1024,
> pub_key=0x7fffffffef38, pri_key=0x7fffffffef40) at generatekey.c:25
> #16 0x0000000010004b78 in main () at generatekey.c:74
> 
> I am compiling both 1.0.2j and 3.0 with same configure command and on
> the same machine(i.e. RHEL 7).  
> 
> Any suggestion or pointer would be highly helpful.

-- 
Tomáš Mráz, OpenSSL