endless loop in probable_prime

[Ronny Meeus]

Op do 18 jun. 2020 om 20:01 schreef Matt Caswell <matt at openssl.org>:
>
>
>
> On 18/06/2020 17:27, Ronny Meeus wrote:
> > Op do 18 jun. 2020 om 18:13 schreef Salz, Rich <rsalz at akamai.com>:
> >>
> >>>    BN_bin2bn assumes that the size of a BN_ULONG (the type of a bn->d) is
> >>     BN_BYTES. You have already told us that sizeof(*d) is 4. So BN_BYTES
> >>     should also be 4. If BN_BYTES is being incorrectly set to 8 on your
> >>     platform then that would explain the discrepancy. Can you check?
> >>
> >> This seems HIGHLY likely since Ronny said earlier that the same config/toolchain is used for 32bit userspace and 64bit kernel, right?
> >>
> >
> > I used the following print statement so the sizeof is actual of the *d
> > and not of the pointer :-).
> > printf("BN_num data (size=%d top=%d neg=%d flags=%d, sizeof(*d)=%d)
> > BN_BYTES=%d: ", rnd->dmax, rnd->top, rnd->neg, rnd->flags,
> > sizeof(*rnd->d), BN_BYTES);
> >
> > The output clearly shows that BN_BYTES is 8:
> >
> > BN_num data (size=24 top=24 neg=0 flags=13, sizeof(*d)=4) BN_BYTES=8:
> > efe838eb2cf627a7cf1944d5969e17602474f949d4e04f33263e49cdc9917b5f4f71f4f27eb06b2f41930dbac791ded7fae69fa604ec65686b412ef048e91cfd8c976e74ff237112406371b6cb2770588328f8db400718366665b6bca733a19c
> >
> > I think we are getting close to the root-cause here ...
>
> Ok, this is clearly why you're getting strange results. The question is
> why is sizeof(*d) different to the value of BN_BYTES?
>
> What Configure target did you use to build for this platform?

We have identified the issue.
We use BR to compile our projects and it is a bug in the buildroot scripts.
This was the code before the patch
        default "linux-generic64 no-asm"        if BR2_ARCH_IS_64
        default "linux-generic32 no-asm"
resulting in the usage of the "linux-generic64 no-asm" target.

When change the above line in:
        default "linux-generic64 no-asm"        if BR2_ARCH_IS_64 &&
!BR2_MIPS_NABI32
        default "linux-generic32 no-asm"
the target becomes "linux-generic32 no-asm".

Compiling openssl with this solves the issue. We will deliver this
correction to the buildroot community.
Thanks all for the great support!


> Also, take a look at the end of the file include/openssl/opensslconf.h
>
> You should see a section at the end that looks like this:
>
> /* Only one for the following should be defined */
> # define SIXTY_FOUR_BIT_LONG
> # undef SIXTY_FOUR_BIT
> # undef THIRTY_TWO_BIT

For completeness, this was the section in the file:

/* Only one for the following should be defined */
# define SIXTY_FOUR_BIT_LONG
# undef SIXTY_FOUR_BIT
# undef THIRTY_TWO_BIT

> This file is generated, and depending on what Configure has decided is
> appropriate for your platform one of the above 3 should be defined, and
> the other 2 undef'd. What does it show for your platform?
>
> Matt
>