Problems porting Openssl 1.1.1d to zos.

[Stephan Mühlstrasser]

Hello Kevin,

Am 14.04.20 um 10:00 schrieb K Lengauer:
> Dear all,
> 
> I want to add another issue that occurred to me and would appreciate some
> input from others using zOS OpenSSL.
> 
> Calls like "ossl_isascii(c)" such as is done in "a_print.c"  in method "int
> ASN1_PRINTABLE_type(const unsigned char *s, int len)" lead to wrong behavior
> for me on zOS if the input is ASCII (already).

I think your observation is correct. There are multiple places in the 
code where the ossl_... character classification macros are applied to 
codes that are ASCII. I documented a similar problem in the following 
issue on GitHub:

https://github.com/openssl/openssl/issues/11385

> "ossl_isascii" leads to a call to "ossl_ctype_check" with the ASCII mask
> 'CTYPE_MASK_ascii'. However, the issue now occurs in there because inside
> "ossl_ctype_check" the function "ossl_toascii" is called.
> 
> int ossl_ctype_check(int c, unsigned int mask)
> {
>      const int max = sizeof(ctype_char_map) / sizeof(*ctype_char_map);
>      const int a = ossl_toascii(c);
> 
>      return a >= 0 && a < max && (ctype_char_map[a] & mask) != 0;
> }
> 
> "ossl_toascii" does convert the input to ASCII unless it is outside the
> range checked via:
> 
>   if (c < -128 || c > 256 || c == EOF)
> 
> So a wrong conversion occurs when the input is ASCII as int/decimal values
> usually range from32-126, so they are not caught in any way by
> "ossl_toascii". When checking if the input is ASCII which it is (expected
> output '1' == true, is ASCII): the input ASCII chars are converted AGAIN to
> ASCII leading to a wrong/weird output and we get a wrong '0' output
> afterwards in "ossl_ctype_check" as 'a' is not ASCII anymore.
> 
> There would have to be an input check like such that the conversion does not
> take place if the input is already in ASCII. But I don't know if this is
> possible easily. Also the EBCDIC space with integer value  '64' would be
> troublesome...
> 
> Did I miss something crucial or did I make a mistake? If so, please let me
> know.
> 
> My next steps will be to try to refactor the "ossl_ctype_check" to not use
> "ossl_toascii" directly but to have some check beforehand. I am not sure if
> this will work everywhere and also the 'exceptions' such as EBCDIC space and
> so on need to be caught correctly. If somebody has already fixed this issue
> or has other ideas they are most welcome.
> 
> 
> 
> --
> Sent from: http://openssl.6102.n7.nabble.com/OpenSSL-User-f3.html
> 

-- 
Stephan