endless loop in probable_prime

[Viktor Dukhovni]

On Thu, Jun 18, 2020 at 05:10:08PM +0200, Ronny Meeus wrote:

> > What is the "endianness" of this CPU?  Is it big-endian or
> > little-endian?  And does OpenSSL expect the correct one?
> 
> This is a big endian CPU.

And, is OpenSSL configured for the correct endianness?

On Thu, Jun 18, 2020 at 05:06:50PM +0200, Ronny Meeus wrote:

> I instrumented both the "probable_prime" and the "bnrand" functions as
> you can see below and the values actually change (see below).
> But what is strange is that the value of "rnd->size" is only 24 and
> the size of the sizeof(rnd->d) is 4 bytes = 32 bits which results only

If rnd->d is a pointer, you want sizeof(rnd->d[0]).  Or if you
literrally mean "sizeof(sizeof(rnd->d))" than that's expected with a
32-bit address space.  Either way, you're perhaps reporting a misleading
value.

> So there is a problem with the calculation of the size.

Or not.

> It might have to do with the fact that we run on a 64b kernel with a
> 32b user land and that we use only 1 toolchain to compile both the
> kernel and the user-land.

The size of the address space (size of pointers) should not matter,
what matters are the sizes "int", "long", "long long", ...

> probable_prime 0 loop=228 again=114
> bnrand bits=1536 top=1 bottom=1 bytes=192
> data: ff ec 15 94 95 10 ca 68 0f 54 06 0f b9 14 bc da 5d 33 58 4e 3e
                    ----  5 ---
> 35 ca d5 30 c8 33 a5 c1 61 7f d6 00 26 98 45 df 73 34 3c 32 74 89 e2
> da 03 72 20 fe 50 23 85 e7 a0 a6 97 27 38 e8 3e 4d 1d d6 11 66 93 d2
> 01 40 5a 9e 23 31 1f a8 a8 7c 46 c9 eb 79 02 44 9d 9c a7 fe 5e 80 f4
> 0b 68 ec a5 2f 59 59 a3 02 56 6b 16 7a 01 a5 bb 3c c9 28 71 3b f5 44
> 30 a3 00 fb f0 de 65 2a 28 8d 8b 8d 2c 71 84 65 33 a9 73 8d 4e ce d4
> 86 ce 1e 6d d7 2f a4 2d 6c 1c 27 88 0e 2a 0c 09 d0 ff 10 e7 55 7b 55
> a9 34 5c 36 88 c9 03 64 f2 ac ae cc 64 f7 a2 3c 6d 62 72 55 8d 99 69
                 ----  4 ---             ----  3 ---             ---- 
> 20 2e 33 8b b6 21 20 dd 57 ef
  2 ---             ---- 1  ---

> probable_prime BN_num_bits=1473 bits=1536
> BN_num data (size=24 top=24 neg=0 flags=13, sizeof(*d)=4):
> 20dd57f1 9969202e f7a23c6d c90364f2 [...] 9510ca68
  --  1 -- --  2 -- --  3 -- --  4 -- ...   --  5 --

However, clearly the hex output above captures only every other 32-bits
of the actual byte array.  Perhaps there's something wrong with your
compilation tool-chain.

-- 
    Viktor.