endless loop in probable_prime

[Matt Caswell]

On 18/06/2020 17:27, Ronny Meeus wrote:
> Op do 18 jun. 2020 om 18:13 schreef Salz, Rich <rsalz at akamai.com>:
>>
>>>    BN_bin2bn assumes that the size of a BN_ULONG (the type of a bn->d) is
>>     BN_BYTES. You have already told us that sizeof(*d) is 4. So BN_BYTES
>>     should also be 4. If BN_BYTES is being incorrectly set to 8 on your
>>     platform then that would explain the discrepancy. Can you check?
>>
>> This seems HIGHLY likely since Ronny said earlier that the same config/toolchain is used for 32bit userspace and 64bit kernel, right?
>>
> 
> I used the following print statement so the sizeof is actual of the *d
> and not of the pointer :-).
> printf("BN_num data (size=%d top=%d neg=%d flags=%d, sizeof(*d)=%d)
> BN_BYTES=%d: ", rnd->dmax, rnd->top, rnd->neg, rnd->flags,
> sizeof(*rnd->d), BN_BYTES);
> 
> The output clearly shows that BN_BYTES is 8:
> 
> BN_num data (size=24 top=24 neg=0 flags=13, sizeof(*d)=4) BN_BYTES=8:
> efe838eb2cf627a7cf1944d5969e17602474f949d4e04f33263e49cdc9917b5f4f71f4f27eb06b2f41930dbac791ded7fae69fa604ec65686b412ef048e91cfd8c976e74ff237112406371b6cb2770588328f8db400718366665b6bca733a19c
> 
> I think we are getting close to the root-cause here ...

Ok, this is clearly why you're getting strange results. The question is
why is sizeof(*d) different to the value of BN_BYTES?

What Configure target did you use to build for this platform?

Also, take a look at the end of the file include/openssl/opensslconf.h

You should see a section at the end that looks like this:

/* Only one for the following should be defined */
# define SIXTY_FOUR_BIT_LONG
# undef SIXTY_FOUR_BIT
# undef THIRTY_TWO_BIT

This file is generated, and depending on what Configure has decided is
appropriate for your platform one of the above 3 should be defined, and
the other 2 undef'd. What does it show for your platform?

Matt