endless loop in probable_prime

[Ronny Meeus]

Op do 18 jun. 2020 om 18:13 schreef Salz, Rich <rsalz at akamai.com>:
>
> >    BN_bin2bn assumes that the size of a BN_ULONG (the type of a bn->d) is
>     BN_BYTES. You have already told us that sizeof(*d) is 4. So BN_BYTES
>     should also be 4. If BN_BYTES is being incorrectly set to 8 on your
>     platform then that would explain the discrepancy. Can you check?
>
> This seems HIGHLY likely since Ronny said earlier that the same config/toolchain is used for 32bit userspace and 64bit kernel, right?
>

I used the following print statement so the sizeof is actual of the *d
and not of the pointer :-).
printf("BN_num data (size=%d top=%d neg=%d flags=%d, sizeof(*d)=%d)
BN_BYTES=%d: ", rnd->dmax, rnd->top, rnd->neg, rnd->flags,
sizeof(*rnd->d), BN_BYTES);

The output clearly shows that BN_BYTES is 8:

BN_num data (size=24 top=24 neg=0 flags=13, sizeof(*d)=4) BN_BYTES=8:
efe838eb2cf627a7cf1944d5969e17602474f949d4e04f33263e49cdc9917b5f4f71f4f27eb06b2f41930dbac791ded7fae69fa604ec65686b412ef048e91cfd8c976e74ff237112406371b6cb2770588328f8db400718366665b6bca733a19c

I think we are getting close to the root-cause here ...

Ronny