endless loop in probable_prime

Ronny Meeus ronny.meeus at gmail.com
Thu Jun 18 07:46:52 UTC 2020


Hello

we are in the process of upgrading our openssl to version 1.1.1g.
On one of our architectures (Cavium MIPS, running kernel 4.9) we have
an issue in the ssh-keygen tool. It keeps on consuming 100% CPU of 1
core.
On other architectures we do not see the issue at all.

I instrumented the openssl library with some traces and observed that
it keeps on looping in the "probable prime" function.
At the end of the function the "BN_num_bits" check is done and if the
return value is not equal to "bits" it basically starts all over
again.

    }
    if (!BN_add_word(rnd, delta))
        return 0;
    if (BN_num_bits(rnd) != bits) {
        printf("%s BN_num_bits %d %d\n", __FUNCTION__, BN_num_bits(rnd), bits);
        goto again;
    }
    bn_check_top(rnd);
    return 1;
}

I added the print function and the result of the print is as follows:
probable_prime BN_num_bits 1473 1536
This trace keeps on going forever and the values never change.

Any idea what could be the underlying root-cause?

Many thanks and best regards,
Ronny


More information about the openssl-users mailing list