Problems porting Openssl 1.1.1d to zos.

[K Lengauer]

Dear all,

I want to add another issue that occurred to me and would appreciate some
input from others using zOS OpenSSL.

Calls like "ossl_isascii(c)" such as is done in "a_print.c"  in method "int
ASN1_PRINTABLE_type(const unsigned char *s, int len)" lead to wrong behavior
for me on zOS if the input is ASCII (already).

"ossl_isascii" leads to a call to "ossl_ctype_check" with the ASCII mask
'CTYPE_MASK_ascii'. However, the issue now occurs in there because inside
"ossl_ctype_check" the function "ossl_toascii" is called. 

int ossl_ctype_check(int c, unsigned int mask)
{
    const int max = sizeof(ctype_char_map) / sizeof(*ctype_char_map);
    const int a = ossl_toascii(c);

    return a >= 0 && a < max && (ctype_char_map[a] & mask) != 0;
}

"ossl_toascii" does convert the input to ASCII unless it is outside the
range checked via:

 if (c < -128 || c > 256 || c == EOF)

So a wrong conversion occurs when the input is ASCII as int/decimal values
usually range from32-126, so they are not caught in any way by
"ossl_toascii". When checking if the input is ASCII which it is (expected
output '1' == true, is ASCII): the input ASCII chars are converted AGAIN to
ASCII leading to a wrong/weird output and we get a wrong '0' output
afterwards in "ossl_ctype_check" as 'a' is not ASCII anymore. 

There would have to be an input check like such that the conversion does not
take place if the input is already in ASCII. But I don't know if this is
possible easily. Also the EBCDIC space with integer value  '64' would be
troublesome...

Did I miss something crucial or did I make a mistake? If so, please let me
know.

My next steps will be to try to refactor the "ossl_ctype_check" to not use
"ossl_toascii" directly but to have some check beforehand. I am not sure if
this will work everywhere and also the 'exceptions' such as EBCDIC space and
so on need to be caught correctly. If somebody has already fixed this issue
or has other ideas they are most welcome.



--
Sent from: http://openssl.6102.n7.nabble.com/OpenSSL-User-f3.html