[openssl-users] [openssl-dev] Failed TLSv1.2 handshake with error 67702888--bad signature

[Kurt Roeckx]

On Sat, Feb 27, 2016 at 06:23:43PM +0000, Dr. Stephen Henson wrote:
> On Sat, Feb 27, 2016, Nounou Dadoun wrote:
> 
> > Thanks for the response,
> > 
> > I'm not sure what you're saying here other than TLS 1.2 client cert auth
> > processing is different from TLS x (where x<1.2); I would assume that the
> > range of mechanisms would expand to include more robust algorithms as time
> > goes on.  However, here something is breaking backward compatibility with a
> > client certificate that is still valid and otherwise correct as far as I can
> > tell.  Our (many) deployed clients support TLSv1.2 and this certificate is
> > widely distributed - we are trying to upgrade the server side from TLSv1 to
> > TLSv1.2 and this appears to be a blocker.
> > 
> > Any recommendations?  I'm still not clear what it is about this certificate
> > that fails in TLSv1.2; I would define a server callback for the certificate
> > verification (I might experiment with that anyway) but it's not clear to me
> > that it would call the callback if the signature is failing.
> > 
> 
> Well which version of OpenSSL is being used by the server and what software is
> the client using?

>From looking at the packet captures, I would guess that both sides
are OpenSSL.

> When client auth is enabled the client presents its certificate chain to the
> server. It also signs some data using the private key corresponsing to the
> public key in the client certificate and the server verifies that signature.
> 
> It looks like it is this latter signature which is causing the problems and
> nothing to do with the certificates (though they have some odd fields in there
> OpenSSL wouldn't reject them). As I mentioned the type of signature a client
> generates during client auth changed in TLS 1.2. It is possible that the
> client is sending an invalid signature which the server is rejecting while
> the different form used for TLS 1.1 and earlier is fine.

It's using RSA/SHA512 (0x0601), which the server offered.


Kurt