How to fix "OpenSSL failed - error:0A000086:SSL routines::certificate verify failed"

[von Oheimb, David]

Certificate chain building, which is part of cert chain validation, is notoriously hard to debug and get right because there are so many implicit things that may go wrong.
This holds in particular for TLS connection setup, but also for most other use cases like checking at application level the signature on a document or message received.
Unfortunately, the error messages by OpenSSL and similar libraries when the chain construction fails are typically not very helpful to find out where the problem(s) is/are.

For this reason, I proposed a chain building trace feature for OpenSSL at https://github.com/openssl/openssl/pull/18761 about half a year ago.
It shows which candidate certificates are tried at which points in which order, and at which points which candidate gets ruled out for which reasons.
Yet the further handling got stuck because it requires some code refactoring, and the respective PR https://github.com/openssl/openssl/pull/18762 is still awaiting review.
Nevertheless, those who are able to compile OpenSSL themselves from a specific branch, namely the branch given in PR 18761<https://github.com/siemens/openssl/tree/x509_trace_chain_building>, may already make use of this.

David

On Sun, 2022-12-18 at 19:46 +0000, Michael Wojcik via openssl-users wrote:
Please send replies to the list, not to me directly, as a courtesy to other people reading the thread and to allow them to reply to your questions.

From: Pierre-Luc Boily <pierreluc.boily at gmail.com<mailto:pierreluc.boily at gmail.com>>
Sent: Friday, 16 December, 2022 21:10

Your questions suggest you're not a TLS expert.

I humbly confess that I am a total beginner!.  Reading the books you proposed is absolutely
something we will do.  But before becoming an expert on this topic (this will take some time
I guess), I'd like to have my c++ client working well with openSSL for demo purposes

I understand, but it may be faster to at least gain more understanding of how TLS works in common cases than spend a lot of time trying to get your demonstration working.

Our project needs a key/certificate because it uses WebRTC on browsers such as chrome and
edge and  WebRTC is not available on a non-secure browser. We also need a https/websocket
server.

Is this also the server for the WebRTC connection? If not, do you have control of that server, or is it a third-party server?

As a first step, I quickly created an https/websocket server with nodejs, and a React front end.
I created my own SSL Certificate Authority ...  I have a .key, a .crt and a .pem(my root certificate).

Note that ".key" and ".crt" are generic file extensions which doesn't tell us anything about the file format. A ".key" file might be PKCS#8, possibly PEM-encoded, or various other things. Most often a .crt is a DER-encoded certificate in PEM format (so a Base64-encoded representation of the DER data, with PEM header/footer lines, and possibly other text which will be ignored by decoders), but it could be binary DER or something else.

Presumably you have a key for the root certificate you created, and a key for your entity (server) certificate. Are there any intermediate certificates for your CA? Viktor mentioned this in his reply but it may not be clear to you. Conventional CAs do not sign entity certificates with their root certificates. They sign an intermediate certificate, which may be used to sign entity certificates or other intermediates.

The "certificate chain" that an entity sends should usually be its entity certificates plus any required intermediates, and optionally the root. (Sending the root doesn't help the peer validate the certificate, because the peer has to already trust the root certificate; but it can help with problem investigation and resolution.)

  I added the .pem into windows, and .key and .crt are used in my nodejs server and on my html server as well.

An entity certificate identifies the entity. Do you know your two servers can both use that certificate? Are they running on the same host? Or is it a wildcard certificate?

The second step is that we have a c++ webrtc client and this client has to talk with the nodejs
https/websocket server.  I thought that using the same key/crt would do the trick.  But I have
the error mentioned in my first email.

Without seeing the certificate chain and knowing how the client is trying to connect, we can't do much to diagnose this. Viktor mentioned some common problems. Since you control the servers, my guess is that SNI is not at issue, but telling OpenSSL what you're connecting to and ensuring that matches what the certificate identifies is a common problem.

The client has to receive a certificate that matches the name it tells OpenSSL it expects for the server. That means the certificate needs to have a Subject Alternative Name extension that matches what the client tells OpenSSL. (If the certificate lacks Subject Alternative Names, the CN of the Subject DN is used, but that's deprecated.)

Then there are quite a number of other checks which are performed, including building the chain to a trust anchor (root). Your application tells OpenSSL what roots it trusts, and OpenSSL needs to be able to create a chain of valid certificates from the server's entity certificate to one of those roots. Each certificate along the chain has to be not only the appropriate certificate (i.e. sign the one before and be signed by the one after, as determined by subject/issuer names or key IDs *and* by a valid signature), but also be valid and have the required key-usage values and so on. The full set of rules is large.

I have 4 things I am wondering :
1. The interface I use, IXWebSockect, has SSL_OP_NO_TLSv1_3 defined.  If I understand, it means
OpenSSL TLS1.3.  Is this correct?  I am suspicious...

TLSv1.3 is not an OpenSSL thing; it's a public standard. I'm not sure what you mean by "defined". SSL_OP_NO_TLSv1_3 is an object-type macro defined by the OpenSSL API, which a caller can use to disable TLSv1.3 support for a connection. Is the API you're calling actually passing that to OpenSSL (e.g. with SSL_CTX_set_options)?

In any case, it's unlikely your server only supports TLSv1.3, and the error you're getting doesn't indicate a lack of matching TLS protocol versions, so this doesn't appear to be an issue.

2. No matter the path of the key/crt I give to OpenSSL, I have the same error message.  If the path doesn't exist,
should I expect an error message like "file not found"?  If yes, my problem could be elsewhere...

This is the server certificate and key? Why are you specifying those to OpenSSL at all? The client does not programmatically set anything regarding the server's certificate (which it normally doesn't know until the server sends it), and should not have a copy of the server's key.

3. You used the terminology "collection of trust anchors".  Is this the .pem, .key and .crt?

In your case, it's just what's in the .pem file: the root for your test CA. Trust anchors are the roots you trust to "anchor" (terminate) a chain, and possibly some intermediates that chain to some of those roots so you can verify certificates that aren't sent with their complete chains.

4. If SSL_VERIFY_PEER is the value expected, I guess "i" shall be 1.  So, I decided to check why i == 0,
and I found this in x509_vfy.c line 206:

In my opinion, debugging into the OpenSSL implementation is very much the wrong way to go. The OpenSSL implementation is not particularly easy to understand, and you really want a thorough understanding of TLS, PKIX, and related matters before trying to figure out what it's doing and why.

...

And build_chain(ctx) bring me there (line 3166, x509_vfy.c) :

Have you confirmed (in the debugger) that this is the point at which OpenSSL decides to reject the peer certificate? Guessing at causes by looking at the code is going to be extremely inefficient.

"Untrust mimic" sounds suspicious!

"Suspicious" describes essentially every failed certificate check. That's pretty much the entire point of TLS, and indeed all security mechanisms: to be suspicious.

--
Michael Wojcik
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mta.openssl.org/pipermail/openssl-users/attachments/20221219/978d73ee/attachment-0001.htm>