[openssl-dev] [openssl.org #3751] Undefined behavior invoked in aes_core.c
Bernd Edlinger via RT
rt at openssl.org
Mon Mar 16 14:21:25 UTC 2015
Hi,
This gets reported by GCC-5.0.0 with -fsanitize=undefined in OpenSSL 1.0.0m 5 Jun 2014:
aes_core.c:1144:30: runtime error: left shift of 136 by 24 places cannot be represented in type 'int'
aes_core.c:1151:30: runtime error: left shift of 158 by 24 places cannot be represented in type 'int'
aes_core.c:1137:30: runtime error: left shift of 239 by 24 places cannot be represented in type 'int'
aes_core.c:1130:30: runtime error: left shift of 139 by 24 places cannot be represented in type 'int'
when I look at these lines, I see the following (repeated 4 times):
s0 =
(Td4[(t0 >> 24) ] << 24) ^
(Td4[(t3 >> 16) & 0xff] << 16) ^
(Td4[(t2 >> 8) & 0xff] << 8) ^
(Td4[(t1 ) & 0xff]) ^
rk[0];
and
static const u8 Td4[256] = {
0x52U, 0x09U, 0x6aU, 0xd5U, 0x30U, 0x36U, 0xa5U, 0x38U, ...
I assume u8 means unsigned char.
GCC converts the u8 to int before the shift left 24.
However, this is undefined behavior in C99/C11, and defined behavior in C++11.
Quoting C99 6.5.7/4:
"The result of E1 << E2 is E1 left-shifted E2 bit positions; vacated bits are filled with zeros. If E1 has an unsigned type, the value of the result is E1 × 2 ^ E2, reduced modulo one more than the maximum value representable in the result type. If E1 has a signed type and nonnegative value, and E1 × 2 ^ E2 is representable in the result type, then that is the resulting value; otherwise, the behavior is undefined."
See also https://gcc.gnu.org/bugzilla/show_bug.cgi?id=65435
With kind regards,
Bernd Edlinger
More information about the openssl-dev
mailing list