[openssl-dev] [openssl.org #4552] Bug report: hex string is too long, problem in set_hex()?

[David-E Young via RT]

Classification: Public
OS: SUSE Linux Enterprise Server 11 SP2  (x86_64)
OpenSSL: versions 1.0.1m, 1.1.0-pre5

Using this command sequence:

echo WuNhPwuWAOiG86RfO4A5jITR9WZ+kF1L+iBgGPQJ4dEJk8Sxiqb014bJsEGDbCfk | $ssl/bin/openssl enc -aes128 -d -a -iv 57fd56a7e47b9482096ab4707ca9d383 -K 617364696f69636a61736f3932336b3b0a

we get different behavior between version 1.0.1m and later versions. With version 1.0.1m the above sequence works fine. Using later versions, including 1.1.0-pre5, we get:

hex string is too long
invalid hex key value

The trouble is in set_hex():

static int set_hex(char *in, unsigned char *out, int size)
{
    ...
    if (n > (size * 2)) {
       // hex string is too long
    ...
    }
}

If the size multiplier is changed to, say, 4, then the problem goes away with no apparent ill effects. Reading the code for set_hex() and its caller, it does not appear that the size multiplier is related to a buffer size or some other limitation.

I've read that the "hex string is too long" issue is a bug in OpenSSL that was supposed to be fixed in the 1.0.2 branch, but that appears not to be the case. Please let us know if we've missed something or if this issue is in fact an OpenSSL bug.

Kind Regards,

David



---
This communication may contain confidential and/or privileged information. If you are not the intended recipient (or have received this communication in error) please notify the sender immediately and destroy this communication. Any unauthorized copying, disclosure or distribution of the material in this communication is strictly forbidden.

Deutsche Bank does not render legal or tax advice, and the information contained in this communication should not be regarded as such.

-- 
Ticket here: http://rt.openssl.org/Ticket/Display.html?id=4552
Please log in as guest with password guest if prompted