[openssl-users] PKI encryption failing on 32-bit Solaris

[Perrow, Graeme]

I have a small test program (source attached) that does a very simple PKI encrypt / decrypt. This program works on Windows, Linux, and Solaris (64-bit) but fails if I run a 32-bit version on Solaris 10. Solaris 11 is fine.

If I use "./config -kPIC -m32 -xarch=sparc" to build OpenSSL, I get a crash in bn_mul_mont_t4_32. I added "no-asm" and it no longer crashes but I get this error output:

OSSL error
4275158204:error:0407109F:rsa routines:RSA_padding_check_PKCS1_type_2:pkcs decoding error:rsa_pk1.c:272:

End OSSL error
decrypt failed

I also tried adding -d to the config line and then the program succeeds though is very verbose.

Could this be a bug in OpenSSL, or am I configuring it incorrectly, or is there a bug in my code?

Thanks for any insight
Graeme Perrow

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mta.openssl.org/pipermail/openssl-users/attachments/20161106/165ccfd6/attachment.html>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: pkiencrypt.cpp
URL: <http://mta.openssl.org/pipermail/openssl-users/attachments/20161106/165ccfd6/attachment.ksh>