[openssl-users] [openssl-dev] Failed TLSv1.2 handshake with error 67702888--bad signature

[Nounou Dadoun]

That gives me something to work with, the server is using openssl 1.0.2d, the client openssl 1.0.1h

I'd actually had an earlier interop problem between them (which I had a mailing list discussion about here: http://openssl.6102.n7.nabble.com/Failed-TLSv1-2-handshake-td61528.html#a61630 ) where server and client were negotiating TLSv1.2 with TLS_RSA_WITH_AES_256_GCM_SHA384 and the handshake failed with the error "decryption failed or bad record mac" - (that scenario was not doing mutual authentication) and my eventual workaround was to disable AESGCM from the cipher list  - which got things going again - with the intention of figuring out what the interop issue was later.  There's a packet capture of a sample failed exchange and more information about that overall scenario in that email thread.

>From that earlier thread:

----
BTW, the client uses OpenSSL 1.0.1h, in boost it's configured with: 
SslContextPtr_t pSslContext = std::make_shared<Net::SslContext>(pIoService, boost::asio::ssl::context::sslv23); 

With no other options. 
Client-side configure header is: 

VERSION=1.0.1h 
MAJOR=1 
MINOR=0.1 
SHLIB_VERSION_NUMBER=1.0.0 
SHLIB_VERSION_HISTORY= 
SHLIB_MAJOR=1 
SHLIB_MINOR=0.0 
SHLIB_EXT= 
PLATFORM=VC-WIN64A 
OPTIONS=enable-rsa enable-md5 enable-sha enable-sha1 enable-dsa enable-des enable-rc4 enable-dh no-bf no-camellia no-cast no-ec_nistp_64_gcc_128 no-gmp no-idea no-jpake no-krb5 no-md2 no-md4 no-mdc2 no-rc2 no-rc5 no-rfc3779 no-ripemd no-sctp no-shared no-store no-zlib no-zlib-dynamic 
CONFIGURE_ARGS=no-idea no-rc5 enable-rsa no-md2 no-md4 enable-md5 enable-sha enable-sha1 enable-dsa no-ripemd enable-des enable-rc4 no-rc2 no-bf no-camellia no-cast no-mdc2 enable-dh VC-WIN64A 
SHLIB_TARGET= 

And as I mentioned earlier, the server uses OpenSSL 1.0.2d which makes for an unusual interop problem! 
---

The openssl versions and ciphers are the same here except now the context has been changed to add mutual authentication (this is a configuration/control channel and we want to make sure that only authenticated clients can get in).   After the cipher change the negotiated cipher is now TLS_RSA_WITH_AES_256_CBC_SHA256.

The fact that they're both fairly recent versions of openssl negotiating TLSv1.2 (as I noted with a one-line change on the server side) makes things very mysterious.  Given the wide distributions of both these versions, I'm a little surprised that our setup seems to be having these fundamental issues in what I expect should be a very common scenario.  

Unfortunately at this point, I'm limited to making changes on the server and that's where my investigation needs to be.    Any breadcrumbs you could drop to help with that investigation would be much appreciated, (and I'd be happy to supply any artifacts that might help track down the issue(s)) thanks ... N

________________________________________
From: openssl-users [openssl-users-bounces at openssl.org] on behalf of Dr. Stephen Henson [steve at openssl.org]
Sent: February 27, 2016 10:23 AM
To: openssl-users at openssl.org
Subject: Re: [openssl-users] [openssl-dev] Failed TLSv1.2 handshake with error 67702888--bad signature

On Sat, Feb 27, 2016, Nounou Dadoun wrote:

> Thanks for the response,
>
> I'm not sure what you're saying here other than TLS 1.2 client cert auth
> processing is different from TLS x (where x<1.2); I would assume that the
> range of mechanisms would expand to include more robust algorithms as time
> goes on.  However, here something is breaking backward compatibility with a
> client certificate that is still valid and otherwise correct as far as I can
> tell.  Our (many) deployed clients support TLSv1.2 and this certificate is
> widely distributed - we are trying to upgrade the server side from TLSv1 to
> TLSv1.2 and this appears to be a blocker.
>
> Any recommendations?  I'm still not clear what it is about this certificate
> that fails in TLSv1.2; I would define a server callback for the certificate
> verification (I might experiment with that anyway) but it's not clear to me
> that it would call the callback if the signature is failing.
>

Well which version of OpenSSL is being used by the server and what software is
the client using?

When client auth is enabled the client presents its certificate chain to the
server. It also signs some data using the private key corresponsing to the
public key in the client certificate and the server verifies that signature.

It looks like it is this latter signature which is causing the problems and
nothing to do with the certificates (though they have some odd fields in there
OpenSSL wouldn't reject them). As I mentioned the type of signature a client
generates during client auth changed in TLS 1.2. It is possible that the
client is sending an invalid signature which the server is rejecting while
the different form used for TLS 1.1 and earlier is fine.

Steve.
--
Dr Stephen N. Henson. OpenSSL project core developer.
Commercial tech support now available see: http://www.openssl.org
--
openssl-users mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users