[openssl-dev] ecp_nistz256 is_one is too liberal with what it considers to be one

[Andy Polyakov]

> Please see the attached program and consider the following change:
> 
> ```
>    if (P256_LIMBS == 8) {
>      res |= a[4] ^ ONE[4];
>      res |= a[5] ^ ONE[5];
>      res |= a[6] ^ ONE[6];
> +    res |= a[7] ^ ONE[7];
>    }

It's not actually a coincidence that it ends with a[6]. If you have
close look at ecp_nistz256_is_affine_G, you'll see that it also check
for generator->Z.top being P256_LIMBS - P256_LIMBS / 8, or 7[!] on
32-bit platform. I.e. we can't assume that a[7] is actually an
initialized value. Quite contrary actually, because there is
configuration flag that will put some junk there on purpose. But yes, it
contradicts second usage case of is_one... Which should be complemented
with additional

    if (P256_LIMBS == 8 && r->Z_is_one)
        r->Z_is_one = (bn_get_top(r->Z) == 7);