endless loop in probable_prime

[Matt Caswell]

On 18/06/2020 08:46, Ronny Meeus wrote:
> Hello
> 
> we are in the process of upgrading our openssl to version 1.1.1g.
> On one of our architectures (Cavium MIPS, running kernel 4.9) we have
> an issue in the ssh-keygen tool. It keeps on consuming 100% CPU of 1
> core.
> On other architectures we do not see the issue at all.
> 
> I instrumented the openssl library with some traces and observed that
> it keeps on looping in the "probable prime" function.
> At the end of the function the "BN_num_bits" check is done and if the
> return value is not equal to "bits" it basically starts all over
> again.
> 
>     }
>     if (!BN_add_word(rnd, delta))
>         return 0;
>     if (BN_num_bits(rnd) != bits) {
>         printf("%s BN_num_bits %d %d\n", __FUNCTION__, BN_num_bits(rnd), bits);
>         goto again;
>     }
>     bn_check_top(rnd);
>     return 1;
> }
> 
> I added the print function and the result of the print is as follows:
> probable_prime BN_num_bits 1473 1536

Hmm. That is very suspicious. I wonder if the generated number in `rnd`
is actually different each time? Can you check?

Matt



> This trace keeps on going forever and the values never change.
> 
> Any idea what could be the underlying root-cause?
> 
> Many thanks and best regards,
> Ronny
>