endless loop in probable_prime

[Guido Vranken]

I think this could be an issue with the system's /dev/urandom or entropy,
as I've observed similar infinite loops in BN_prime when I changed OpenSSL
code to always return the same sequence of bytes from its PRNG (for testing
purposes). It could also be a genuine bug in OpenSSL, or both. I'll let
others comment on that.

On Thu, Jun 18, 2020 at 9:47 AM Ronny Meeus <ronny.meeus at gmail.com> wrote:

> Hello
>
> we are in the process of upgrading our openssl to version 1.1.1g.
> On one of our architectures (Cavium MIPS, running kernel 4.9) we have
> an issue in the ssh-keygen tool. It keeps on consuming 100% CPU of 1
> core.
> On other architectures we do not see the issue at all.
>
> I instrumented the openssl library with some traces and observed that
> it keeps on looping in the "probable prime" function.
> At the end of the function the "BN_num_bits" check is done and if the
> return value is not equal to "bits" it basically starts all over
> again.
>
>     }
>     if (!BN_add_word(rnd, delta))
>         return 0;
>     if (BN_num_bits(rnd) != bits) {
>         printf("%s BN_num_bits %d %d\n", __FUNCTION__, BN_num_bits(rnd),
> bits);
>         goto again;
>     }
>     bn_check_top(rnd);
>     return 1;
> }
>
> I added the print function and the result of the print is as follows:
> probable_prime BN_num_bits 1473 1536
> This trace keeps on going forever and the values never change.
>
> Any idea what could be the underlying root-cause?
>
> Many thanks and best regards,
> Ronny
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mta.openssl.org/pipermail/openssl-users/attachments/20200618/e488326e/attachment.html>