Problems porting Openssl 1.1.1d to zos.

[K Lengauer]

First of all, thanks Michael Wojcik for your answer regarding the datasets. I
was able to get it working.

In the meantime I got the whole build done and am working on my tests. One
thing that I noticed recently is a wrong certificate X509 name output that
happens because of the following code section in "x509_obj.c":

#ifdef CHARSET_EBCDIC
        if (type == V_ASN1_GENERALSTRING ||
            type == V_ASN1_VISIBLESTRING ||
            type == V_ASN1_PRINTABLESTRING ||
            type == V_ASN1_TELETEXSTRING ||
            type == V_ASN1_IA5STRING) {
            if (num > (int)sizeof(ebcdic_buf))
                num = sizeof(ebcdic_buf);
            ascii2ebcdic(ebcdic_buf, q, num);
            q = ebcdic_buf;
        }
#endif

On zOS during my tests the input type I have is "V_ASN1_UTF8STRING" for my
certificates. Thus, the ascii2ebcdic conversion never happens but in the
following lines on code which are executed the causes an issue as ASCII
instead of EBCDIC chars are treated with "os_toascii".

#ifndef CHARSET_EBCDIC
            if ((q[j] < ' ') || (q[j] > '~'))
                l2 += 3;
#else
            if ((os_toascii[q[j]] < os_toascii[' ']) ||
                (os_toascii[q[j]] > os_toascii['~']))
                l2 += 3;
#endif

This finally leads to weird behavior with the comparison to ' ' (space) and
'~' and causes the output to be hex chars due to the following code section
in "x509_obj.c":
            n = os_toascii[q[j]];
            if ((n < os_toascii[' ']) || (n > os_toascii['~'])) {
                *(p++) = '\\';
                *(p++) = 'x';
                *(p++) = hex[(n >> 4) & 0x0f];
                *(p++) = hex[n & 0x0f];
            } else
                *(p++) = q[j];

Now, I am aware that there are several EBCDIC issues as OpenSSL is to my
knowledge not currently tested for zOS (see also:
https://github.com/openssl/openssl/issues/4154).
If I add "type == V_ASN1_UTF8STRING" to the list of allowed types I was able
to generate a human readable output. 

#ifdef CHARSET_EBCDIC
        if (type == V_ASN1_GENERALSTRING ||
            type == V_ASN1_VISIBLESTRING ||
            type == V_ASN1_PRINTABLESTRING ||
            type == V_ASN1_TELETEXSTRING ||
            type == V_ASN1_UTF8STRING ||
            type == V_ASN1_IA5STRING) {
            if (num > (int)sizeof(ebcdic_buf))
                num = sizeof(ebcdic_buf);
            ascii2ebcdic(ebcdic_buf, q, num);
            q = ebcdic_buf;
        }
#endif

However, I wanted to ask for any concerns and other inputs here. Am I
missing anything major here?
As UTF8 is a superset of ASCII there might be other issues with this change
that I have overlooked so far.




--
Sent from: http://openssl.6102.n7.nabble.com/OpenSSL-User-f3.html