[openssl-dev] [openssl.org #3852] bn_gfm2.c: in BN_GF2m_mod_arr() a check is optimized out

[Kurt Cancemi via RT]

Found by the https://github.com/xiw/stack tool and then I checked the
generated asm (gcc and clang) to confirm.

In the check "if (d0 && tmp_ulong)" tmp_ulong always evaluates to true
because the compiler optimizes out the tmp_ulong value to true because
(tmp_ulong = zz >> d1;) zz >> d1 has according to the compiler (LLVM)
a logical right-shift overflow. The commit "don't write beyond buffer
" (c237de0) added this check.

---
Kurt Cancemi
https://www.x64architecture.com