[openssl-dev] [openssl.org #3852] bn_gfm2.c: in BN_GF2m_mod_arr() a check is optimized out

[Kurt Cancemi via RT]

I don't think its optimizing it out I agree with you, but your
suggested change resolved the error so Wim was right about it being
undefined behaviour, and the tool which uses clang 3.4 was warning us
about that.
I added my own debugging statement and ran ectest and it is indeed
undefined behaviour (though it doesn't matter much as the results not
used as WIm stated) before your patch:

printf("DBG: zz=" BN_DEC_FMT1 " d0=%d\n", zz, d0);
Example Output line (before your patch):
DBG: zz=1 d0=1 (UB)
Output line (after your patch):
DBG: zz=89619874872 d0=7


On Tue, May 19, 2015 at 4:04 PM, Andy Polyakov via RT <rt at openssl.org> wrote:
>
> The commit intention is actually sane. This kind of brings us back to
> question why was it optimized away? BTW, which compiler version is it
> and what are your optimization flags? I can't reproduce the problem with
> clang 3.6 and -O3. I mean I don't see that check for tmp_ulong is
> optimized away. In either case, as intention is sane, the only possible
> modification would be
>
>         if (d0 && (tmp_ulong = zz >> d1))
>
> So that right shift is not attempted if d0 is zero.
>
>