[openssl-dev] [openssl.org #3852] bn_gfm2.c: in BN_GF2m_mod_arr() a check is optimized out

[Kurt Cancemi via RT]

>>>> Found by the https://github.com/xiw/stack tool and then I checked the
>>>> generated asm (gcc and clang) to confirm.
>>>>
>>>> In the check "if (d0 && tmp_ulong)" tmp_ulong always evaluates to true
>>>> because the compiler optimizes out the tmp_ulong value to true because
>>>> (tmp_ulong = zz >> d1;) zz >> d1 has according to the compiler (LLVM)
>>>> a logical right-shift overflow.
>>> What's right-shift overflow? In either case, are you sure about it being
>>> optimized away because it always evaluates to true? Thing is that if
>>> tmp_ulong is 0, then xor-ing with it won't have effect on result. I mean
>>> check for d0 alone would actually produce same outcome, wouldn't it?
>>
>> Right-shifting by *equal to* or more than the width of the word produces undefined results in C (just like left-shifting); one expects that it produces zero, but the language spec says it’s undefined.
>
> I understand that. I don't understand why would one call it *overflow*.

Now that I took a look at the tools source, it probably wasn't an
overflow the tool just detected
that the shr instruction was changed and for whatever reason calls it
an overflow no mater the change,
I should have taken a look at the tools source first. But the original
commits intention should be checked
out.