endless loop in probable_prime

[Ronny Meeus]

Op do 18 jun. 2020 om 18:27 schreef Ronny Meeus <ronny.meeus at gmail.com>:
>
> Op do 18 jun. 2020 om 18:13 schreef Salz, Rich <rsalz at akamai.com>:
> >
> > >    BN_bin2bn assumes that the size of a BN_ULONG (the type of a bn->d) is
> >     BN_BYTES. You have already told us that sizeof(*d) is 4. So BN_BYTES
> >     should also be 4. If BN_BYTES is being incorrectly set to 8 on your
> >     platform then that would explain the discrepancy. Can you check?
> >
> > This seems HIGHLY likely since Ronny said earlier that the same config/toolchain is used for 32bit userspace and 64bit kernel, right?
> >
>
> I used the following print statement so the sizeof is actual of the *d
> and not of the pointer :-).
> printf("BN_num data (size=%d top=%d neg=%d flags=%d, sizeof(*d)=%d)
> BN_BYTES=%d: ", rnd->dmax, rnd->top, rnd->neg, rnd->flags,
> sizeof(*rnd->d), BN_BYTES);
>
> The output clearly shows that BN_BYTES is 8:
>
> BN_num data (size=24 top=24 neg=0 flags=13, sizeof(*d)=4) BN_BYTES=8:
> efe838eb2cf627a7cf1944d5969e17602474f949d4e04f33263e49cdc9917b5f4f71f4f27eb06b2f41930dbac791ded7fae69fa604ec65686b412ef048e91cfd8c976e74ff237112406371b6cb2770588328f8db400718366665b6bca733a19c
>
> I think we are getting close to the root-cause here ...
>

I adapted the code so that the BN_BYTES is 4 in all cases (just a hack).
Now the issue is gone and the loop is not observed anymore but the
tool still fails to generate the keys:

LD_LIBRARY_PATH=/tmp ssh-keygen -A
ssh-keygen: generating new host keys: RSA

probable_prime 0 loop=0 again=0
bnrand bits=1536 top=1 bottom=1 bytes=192
data: db d2 2c 9e d8 99 08 af d0 f9 da 7c 19 2a 0d 05 94 e2 ba fd f5
60 38 74 81 aa 1a 1c cf 02 70 0d a6 97 51 c6 4a e4 13 98 39 99 7b c4
30 21 97 a0 32 89 3e 2c ff a2 fb 3f b3 bd f9 6b 59 00 f5 48 41 b7 57
cc a5 37 f9 d8 be ba f9 3d f0 6e 86 58 27 fb ae ca 9c 7b 2e 3f 4c 33
16 74 23 b0 c7 d7 74 19 8a ad 3e 5d 76 73 a5 f5 4a 95 d5 b0 da 5f a0
34 c8 ee 57 fb 88 22 be 5a f5 92 6e 47 08 68 e5 21 6c 57 b2 1b 1d 50
c9 89 d8 a3 76 24 b3 2d f3 b6 26 bc 35 33 b7 d0 73 09 8c 1b 2d 90 05
5a 75 24 05 1a de 04 31 ef 5b b1 55 62 33 86 8a 72 3d bc 5b c9 a1 55
e4 6a 4f cd 2e 28 5f 1a 09 25

probable_prime BN_num_bits=1536 bits=1536
BN_num data (size=48 top=48 neg=0 flags=13, sizeof(*d)=4) BN_BYTES=4:
5f1a09734fcd2e28a155e46a3dbc5bc933868a725bb15562de0431ef7524051a2d90055a73098c1b3533b7d0f3b626bc7624b32dc989d8a3b21b1d50e5216c576e470868be5af59257fb8822a034c8eed5b0da5fa5f54a953e5d767374198aad23b0c7d74c3316749c7b2e3f27fbaecaf06e8658bebaf93da537f9d841b757cc5900f548b3bdf96bffa2fb3f32893e2c302197a039997bc44ae41398a69751c6cf02700d81aa1a1cf560387494e2bafd192a0d05d0f9da7cd89908afdbd22c9e

sshkey_generate failed: error in libcrypto
ssh-keygen: generating new host keys: DSA sshkey_generate failed:
error in libcrypto
ssh-keygen: generating new host keys: ECDSA sshkey_generate failed:
memory allocation failed
ssh-keygen: generating new host keys: ED25519

I also have the impression that it takes a very long time for the tool
to complete.
On other CPUs (ARM) the tool finishes much quicker (and successful).


> Ronny