[openssl-dev] BUG - FIPS capable OpenSSL fails to build on Linux PPC64

[Cristi Fati]

Hi all,

I am trying to build a FIPS (2.0.12) capable OpenSSL (1.0.2h) on PPC64
Linux (tried RH5 and SLES12), but it fails.

Here's the config command and output for *openssl-fips*:

*./config no-asm*
Operating system: *ppc64-whatever-linux2*
WARNING! If you wish to build 64-bit library, then you have to
         invoke './Configure linux-ppc64' *manually*.
         You have about 5 seconds to press Ctrl-C to abort.
Auto Configuring fipsonly
Auto Configuring fipsonly
Configuring for linux-ppc
Auto Configuring fipsonly
Configuring for linux-ppc
    no-asm          [option]   OPENSSL_NO_ASM
    no-bf           [option]   OPENSSL_NO_BF (skip dir)
    no-camellia     [option]   OPENSSL_NO_CAMELLIA (skip dir)
    no-cast         [option]   OPENSSL_NO_CAST (skip dir)
    no-ec_nistp_64_gcc_128 [default]  OPENSSL_NO_EC_NISTP_64_GCC_128 (skip
dir)
    no-gmp          [default]  OPENSSL_NO_GMP (skip dir)
    no-idea         [option]   OPENSSL_NO_IDEA (skip dir)
    no-jpake        [experimental] OPENSSL_NO_JPAKE (skip dir)
    no-krb5         [krb5-flavor not specified] OPENSSL_NO_KRB5
    no-md2          [option]   OPENSSL_NO_MD2 (skip dir)
    no-md5          [option]   OPENSSL_NO_MD5 (skip dir)
    no-mdc2         [option]   OPENSSL_NO_MDC2 (skip dir)
    no-rc2          [option]   OPENSSL_NO_RC2 (skip dir)
    no-rc4          [option]   OPENSSL_NO_RC4 (skip dir)
    no-rc5          [option]   OPENSSL_NO_RC5 (skip dir)
    no-rfc3779      [default]  OPENSSL_NO_RFC3779 (skip dir)
    no-ripemd       [option]   OPENSSL_NO_RIPEMD (skip dir)
    no-seed         [option]   OPENSSL_NO_SEED (skip dir)
    no-srp          [forced]   OPENSSL_NO_SRP (skip dir)
    no-ssl2         [forced]   OPENSSL_NO_SSL2 (skip dir)
    no-ssl3         [forced]   OPENSSL_NO_SSL3 (skip dir)
    no-store        [experimental] OPENSSL_NO_STORE (skip dir)
    no-tls1         [forced]   OPENSSL_NO_TLS1 (skip dir)
    no-tlsext       [forced]   OPENSSL_NO_TLSEXT (skip dir)
    no-zlib         [default]
    no-zlib-dynamic [default]
IsMK1MF=0
CC            =gcc
CFLAG         =-DOPENSSL_FIPSCANISTER -fPIC -DOPENSSL_PIC -DOPENSSL_THREADS
-D_REENTRANT -DDSO_DLFCN -DHAVE_DLFCN_H  -DTERMIO -O3 -Wall



and the corresponding command for *openssl*:

*./config fips shared no-asm ${SOME_OTHER_UNIMPORTANT_OPTIONS}*
Operating system: *ppc64-whatever-linux2*
WARNING! If you wish to build 64-bit library, then you have to
         invoke './Configure linux-ppc64' *manually*.
         You have about 5 seconds to press Ctrl-C to abort.
Configuring for linux-ppc
Configuring for linux-ppc
    no-asm          [option]   OPENSSL_NO_ASM
    no-ec_nistp_64_gcc_128 [default]  OPENSSL_NO_EC_NISTP_64_GCC_128 (skip
dir)
    no-gmp          [default]  OPENSSL_NO_GMP (skip dir)
    no-idea         [option]   OPENSSL_NO_IDEA (skip dir)
    no-jpake        [experimental] OPENSSL_NO_JPAKE (skip dir)
    no-krb5         [krb5-flavor not specified] OPENSSL_NO_KRB5
    no-libunbound   [experimental] OPENSSL_NO_LIBUNBOUND (skip dir)
    no-md2          [default]  OPENSSL_NO_MD2 (skip dir)
    no-mdc2         [option]   OPENSSL_NO_MDC2 (skip dir)
    no-rc5          [option]   OPENSSL_NO_RC5 (skip dir)
    no-rfc3779      [default]  OPENSSL_NO_RFC3779 (skip dir)
    no-rsax         [forced]   OPENSSL_NO_RSAX (skip dir)
    no-sctp         [default]  OPENSSL_NO_SCTP (skip dir)
    no-ssl-trace    [default]  OPENSSL_NO_SSL_TRACE (skip dir)
    no-store        [experimental] OPENSSL_NO_STORE (skip dir)
    no-unit-test    [default]  OPENSSL_NO_UNIT_TEST (skip dir)
    no-zlib         [default]
    no-zlib-dynamic [default]
IsMK1MF=0
CC            =gcc
CFLAG         =-fPIC -DOPENSSL_PIC -DOPENSSL_THREADS -D_REENTRANT
-DDSO_DLFCN -DHAVE_DLFCN_H -fPIC *-m32* -DB_ENDIAN -O3 -Wall
-I$(FIPSDIR)/include



As you can see, they both identify the machine in the same way, but
openssl-fips generates 64bit object files and openssl 32bit, and the linker
when trying to build *fips_premain_dso* obviously doesn't like it.
Here's the error:
*${LINKER_PATH}/ld: powerpc:common64 architecture of input file
`${FIPSCANISTER_PATH}/fipscanister.o' is incompatible with powerpc:common
output*



Last versions that worked for me, were openssl-1.0.1p and
openssl-fips-2.0.5, which were both generating 64bit code, and I first
encountered this behavior in openssl-1.0.2f (didn't test the versions
before).

As I see things there are 3 issues:
- 2 minor: The warning in both *openssl* and *openssl-fips* is
misleading/wrong (shouldn't be there)
- 1 major: *openssl* generates 32bit binaries for ppc64.

RH5 build machine details:
*Linux ${HOST_NAME} 2.6.18-92.el5 #1 SMP Tue Apr 29 13:21:29 EDT 2008 ppc64
ppc64 ppc64 GNU/Linux*

The only way to get around this is to instruct openssl to generate 64bit
code (*./Configure linux-ppc64*), as I can't modify any *openssl-fips*
files and still have a FIPS validated result.


Also, as a note: in *openssl-fips* (since version 2.0.6), *Configure* no
longer has the exec permissions. Was that the intent? Probably yes, since
the only way to configure *openssl-fips* is via *config [no-asm].*


Another note: after having everything built I get:
*error 7 at 0 depth lookup:certificate signature failure*
*550858546160:error:04097077:rsa routines:RSA_private_encrypt:wrong
signature length:fips_rsa_sign.c:349:*
*550858546160:error:0D0C5006:asn1 encoding routines:ASN1_item_verify:EVP
lib:a_verify.c:218:*

when trying to validate a cert against its CA, but only on Linux PPC64 and
when FIPS is enabled - validating the same certs on PPC with FIPS off, or
on another Linux (x86_64) with FIPS on, works fine - (I'm talking about the
same *openssl*, *openssl-fips* versions here), but this is another topic,
which I'm going to send a different mail about once I have more details

Regards,
Cristi Fati.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mta.openssl.org/pipermail/openssl-dev/attachments/20160621/5e691730/attachment.html>