[openssl-dev] [openssl.org #3852] bn_gfm2.c: in BN_GF2m_mod_arr() a check is optimized out

[Andy Polyakov via RT]

>>>>> Found by the https://github.com/xiw/stack tool and then I checked the
>>>>> generated asm (gcc and clang) to confirm.
>>>>>
>>>>> In the check "if (d0 && tmp_ulong)" tmp_ulong always evaluates to true
>>>>> because the compiler optimizes out the tmp_ulong value to true because
>>>>> (tmp_ulong = zz >> d1;) zz >> d1 has according to the compiler (LLVM)
>>>>> a logical right-shift overflow.
>>>> ... are you sure about it being
>>>> optimized away because it always evaluates to true? Thing is that if
>>>> tmp_ulong is 0, then xor-ing with it won't have effect on result. I mean
>>>> check for d0 alone would actually produce same outcome, wouldn't it?
> 
> ... the original
> commits intention should be checked
> out.

The commit intention is actually sane. This kind of brings us back to
question why was it optimized away? BTW, which compiler version is it
and what are your optimization flags? I can't reproduce the problem with
clang 3.6 and -O3. I mean I don't see that check for tmp_ulong is
optimized away. In either case, as intention is sane, the only possible
modification would be

	if (d0 && (tmp_ulong = zz >> d1))

So that right shift is not attempted if d0 is zero.