[openssl-users] [openssl-dev] Failed TLSv1.2 handshake with error 67702888--bad signature

[Dr. Stephen Henson]

On Sat, Feb 27, 2016, Nounou Dadoun wrote:

> That gives me something to work with, the server is using openssl 1.0.2d, the client openssl 1.0.1h
> 
> I'd actually had an earlier interop problem between them (which I had a mailing list discussion about here: http://openssl.6102.n7.nabble.com/Failed-TLSv1-2-handshake-td61528.html#a61630 ) where server and client were negotiating TLSv1.2 with TLS_RSA_WITH_AES_256_GCM_SHA384 and the handshake failed with the error "decryption failed or bad record mac" - (that scenario was not doing mutual authentication) and my eventual workaround was to disable AESGCM from the cipher list  - which got things going again - with the intention of figuring out what the interop issue was later.  There's a packet capture of a sample failed exchange and more information about that overall scenario in that email thread.
> 

That might be a problem with SHA384/SHA512. You can configure OpenSSL 1.0.2
server side to not request RSA+SHA384/RSA+SHA512 and see if that helps. If
TLS v1.2 works other than that then it's likely that SHA256 is OK.

See for example SSL_CTX_set1_client_sigalgs_list() et al at:

https://www.openssl.org/docs/manmaster/ssl/SSL_CTX_set1_client_sigalgs.html

Steve.
--
Dr Stephen N. Henson. OpenSSL project core developer.
Commercial tech support now available see: http://www.openssl.org