endless loop in probable_prime

[Matt Caswell]

Great - glad you found it.

Matt

On 19/06/2020 06:44, Ronny Meeus wrote:
> Replying again since it looks like the final mail has not been
> delivered to the newsgroup archive.
> The issue has been found. It was a bug in buildroot passing the wrong target.
> 
> Op do 18 jun. 2020 om 20:33 schreef Ronny Meeus <ronny.meeus at gmail.com>:
>>
>> Op do 18 jun. 2020 om 20:01 schreef Matt Caswell <matt at openssl.org>:
>>>
>>>
>>>
>>> On 18/06/2020 17:27, Ronny Meeus wrote:
>>>> Op do 18 jun. 2020 om 18:13 schreef Salz, Rich <rsalz at akamai.com>:
>>>>>
>>>>>>    BN_bin2bn assumes that the size of a BN_ULONG (the type of a bn->d) is
>>>>>     BN_BYTES. You have already told us that sizeof(*d) is 4. So BN_BYTES
>>>>>     should also be 4. If BN_BYTES is being incorrectly set to 8 on your
>>>>>     platform then that would explain the discrepancy. Can you check?
>>>>>
>>>>> This seems HIGHLY likely since Ronny said earlier that the same config/toolchain is used for 32bit userspace and 64bit kernel, right?
>>>>>
>>>>
>>>> I used the following print statement so the sizeof is actual of the *d
>>>> and not of the pointer :-).
>>>> printf("BN_num data (size=%d top=%d neg=%d flags=%d, sizeof(*d)=%d)
>>>> BN_BYTES=%d: ", rnd->dmax, rnd->top, rnd->neg, rnd->flags,
>>>> sizeof(*rnd->d), BN_BYTES);
>>>>
>>>> The output clearly shows that BN_BYTES is 8:
>>>>
>>>> BN_num data (size=24 top=24 neg=0 flags=13, sizeof(*d)=4) BN_BYTES=8:
>>>> efe838eb2cf627a7cf1944d5969e17602474f949d4e04f33263e49cdc9917b5f4f71f4f27eb06b2f41930dbac791ded7fae69fa604ec65686b412ef048e91cfd8c976e74ff237112406371b6cb2770588328f8db400718366665b6bca733a19c
>>>>
>>>> I think we are getting close to the root-cause here ...
>>>
>>> Ok, this is clearly why you're getting strange results. The question is
>>> why is sizeof(*d) different to the value of BN_BYTES?
>>>
>>> What Configure target did you use to build for this platform?
>>
>> We have identified the issue.
>> We use BR to compile our projects and it is a bug in the buildroot scripts.
>> This was the code before the patch
>>         default "linux-generic64 no-asm"        if BR2_ARCH_IS_64
>>         default "linux-generic32 no-asm"
>> resulting in the usage of the "linux-generic64 no-asm" target.
>>
>> When change the above line in:
>>         default "linux-generic64 no-asm"        if BR2_ARCH_IS_64 &&
>> !BR2_MIPS_NABI32
>>         default "linux-generic32 no-asm"
>> the target becomes "linux-generic32 no-asm".
>>
>> Compiling openssl with this solves the issue. We will deliver this
>> correction to the buildroot community.
>> Thanks all for the great support!
>>
>>
>>> Also, take a look at the end of the file include/openssl/opensslconf.h
>>>
>>> You should see a section at the end that looks like this:
>>>
>>> /* Only one for the following should be defined */
>>> # define SIXTY_FOUR_BIT_LONG
>>> # undef SIXTY_FOUR_BIT
>>> # undef THIRTY_TWO_BIT
>>
>> For completeness, this was the section in the file:
>>
>> /* Only one for the following should be defined */
>> # define SIXTY_FOUR_BIT_LONG
>> # undef SIXTY_FOUR_BIT
>> # undef THIRTY_TWO_BIT
>>
>>> This file is generated, and depending on what Configure has decided is
>>> appropriate for your platform one of the above 3 should be defined, and
>>> the other 2 undef'd. What does it show for your platform?
>>>
>>> Matt
>>>
>