[openssl-dev] PKI encryption failing on 32-bit Solaris

[Perrow, Graeme]

I sent this email to the openssl-users list the other day, but I wonder if openssl-dev might be a better place.

I have a small test program (source attached) that does a very simple PKI encrypt / decrypt. This program works on Windows, Linux, and Solaris (64-bit) but fails if I run a 32-bit version on Solaris 10. Solaris 11 is fine.

If I use "./config -kPIC -m32 -xarch=sparc" to build OpenSSL, I get a crash in bn_mul_mont_t4_32. I added "no-asm" and it no longer crashes but I get this error output:

OSSL error
4275158204:error:0407109F:rsa routines:RSA_padding_check_PKCS1_type_2:pkcs decoding error:rsa_pk1.c:272:

End OSSL error
decrypt failed

I also tried adding -d to the config line to build a debug OpenSSL and then the program succeeds though is very verbose. The fact that a debug OpenSSL works tells me that my code is likely OK.

Could this be a bug in OpenSSL, or am I configuring it incorrectly, or is there a bug in my code?

Thanks for any insight

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mta.openssl.org/pipermail/openssl-dev/attachments/20161107/34882280/attachment.html>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: pkiencrypt.cpp
URL: <http://mta.openssl.org/pipermail/openssl-dev/attachments/20161107/34882280/attachment.ksh>