[openssl-dev] [openssl.org #3751] Undefined behavior invoked in aes_core.c

[Matt Caswell via RT]

On Mon Mar 16 15:21:24 2015, Bernd.Edlinger at Softing.com wrote:
> Hi,
>
> This gets reported by GCC-5.0.0 with -fsanitize=undefined in OpenSSL
> 1.0.0m 5 Jun 2014:
>
> aes_core.c:1144:30: runtime error: left shift of 136 by 24 places
> cannot be represented in type 'int'
> aes_core.c:1151:30: runtime error: left shift of 158 by 24 places
> cannot be represented in type 'int'
> aes_core.c:1137:30: runtime error: left shift of 239 by 24 places
> cannot be represented in type 'int'
> aes_core.c:1130:30: runtime error: left shift of 139 by 24 places
> cannot be represented in type 'int'
>
>
> when I look at these lines, I see the following (repeated 4 times):
>
> s0 =
> (Td4[(t0 >> 24) ] << 24) ^
> (Td4[(t3 >> 16) & 0xff] << 16) ^
> (Td4[(t2 >> 8) & 0xff] << 8) ^
> (Td4[(t1 ) & 0xff]) ^
> rk[0];
>
> and
> static const u8 Td4[256] = {
> 0x52U, 0x09U, 0x6aU, 0xd5U, 0x30U, 0x36U, 0xa5U, 0x38U, ...
>
> I assume u8 means unsigned char.
>
> GCC converts the u8 to int before the shift left 24.
>
> However, this is undefined behavior in C99/C11, and defined behavior
> in C++11.

Hi Bernd

This issue has already been fixed in git for master, 1.0.2 and 1.0.1. See, for
example, commit 8b37e5c14f in the OpenSSL_1_0_2-stable branch.

This will not be fixed in 1.0.0 and 0.9.8 as it is not a security issue and
these branches are only receiving security fixes due to their EOL status.

Thanks for your report,

Matt