How to fix "OpenSSL failed - error:0A000086:SSL routines::certificate verify failed"

[Michael Wojcik]

[Apologies to the list for top-posting and HTML email. I'm in a hurry and don't have time to reformat.]

No idea who "psv sridhar" is – I don't recall ever seeing that name here on openssl-users before – but I don't see anything inappropriate about your message. Ignore him/her.

OpenSSL is telling you that it wasn't able to verify the peer's certificate. That's probably because you haven't loaded the correct collection of trust anchors (root certificates, and possibly intermediate certificates as well).

You ask whether SSL_VERIFY_PEER should be set. Yes, it *must* be set, except in unusual circumstances (e.g. you're using a pre-shared key); otherwise you're vulnerable to MITM interception and have no security under any reasonable threat model.

Your questions suggest you're not a TLS expert. TLS is extremely easy to get wrong (which is why a huge number of applications, particularly in the mobile space, get it wrong), so I would strongly recommend you do some research before proceeding. There are any number of introductions to SSL/TLS online, though personally if one of my teams were starting with TLS I'd require at least one of them read a more substantial introduction, such as Rescorla's /SSL and TLS/ or Ristic's /Bulletproof TLS/.


From: openssl-users <openssl-users-bounces at openssl.org> On Behalf Of Pierre-Luc Boily
Sent: Friday, 16 December, 2022 16:20
To: psv sridhar <psv_sridhar at yahoo.com>
Cc: openssl-users at openssl.org
Subject: Re: How to fix "OpenSSL failed - error:0A000086:SSL routines::certificate verify failed"

I am asking a question regarding OpenSSL.  I thought the mailing list was the place.  I read this on the github page of OpenSSL

    If you have questions about how to use OpenSSL for specific tasks
    or how to solve certain problems you have when using it, you might
    want to ask them on the openssl-users at openssl.org<mailto:openssl-users at openssl.org> mailing list.
    There you can get help from a great community of OpenSSL users,
    not only (but including) the OpenSSL developers. For more information
    about our mailing lists, see
    https://www.openssl.org/community/mailinglists.html.



Le ven. 16 déc. 2022, à 17 h 14, psv sridhar <psv_sridhar at yahoo.com<mailto:psv_sridhar at yahoo.com>> a écrit :
you are sending flooded emails wrongly. stop it.


Thanks and Regards
Sridhar PSV
Phone 571 244-5862


On Friday, December 16, 2022 at 04:08:38 PM CST, Pierre-Luc Boily <pierreluc.boily at gmail.com<mailto:pierreluc.boily at gmail.com>> wrote:


Hello,

Details
OS : WIndows 10
Arch : x64
Compiler : VisualStudio 2017

I have a c++ wss IXWebSocket<https://github.com/machinezone/IXWebSocket> client that tries to connect to a nodejs https/websocket server but the client refuses to connect and returns the error : OpenSSL failed - error:0A000086:SSL routines::certificate verify failed
What I tried
1.       I have a React front end using wss to communicate to my https nodejs server. It works -> This confirms that my key and certificate are valid.
2.       I also tried the same c++ client above, not secured (no wss) connecting to my same nodejs server, but http/websocket (non secure). It works.
So, I had to dig into the OpenSSL code and I found where the error is triggered, see code below.  In my case s->verify_mode is equal to SSL_VERIFY_PEER and i equal to 0 and I don't know if those values are OK or not.

While I was digging into the code, I also realized that SSL_OP_NO_TLSv1_3 is automagically defined for my code.  I feel that it is incorrect.

From statem_clnt.c line 1888:
    if (s->verify_mode != SSL_VERIFY_NONE && i <= 0) {
        SSLfatal(s, ssl_x509err2alert(s->verify_result),
                 SSL_R_CERTIFICATE_VERIFY_FAILED);
        return WORK_ERROR;
    }

Stacktrace:
> libssl-3-x64.dll!tls_post_process_server_certificate(ssl_st
  libssl-3-x64.dll!ossl_statem_client_post_process_message(ss
  libssl-3-x64.dll!read_state_machine(ssl_st * s) Line 675
  libssl-3-x64.dll!state_machine(ssl_st * s, int server) Line
  libssl-3-x64.dll!ossl_statem_connect(ssl_st * s) Line 266
  libssl-3-x64.dll!SSL_do_handshake(ssl_st * s) Line 3937 C
  libssl-3-x64.dll!SSL_connect(ssl_st * s) Line 1760 C
  testWSClient.exe!ix::SocketOpenSSL::openSSLClientHandshake(
  testWSClient.exe!ix::SocketOpenSSL::connect(const std::basi
  testWSClient.exe!ix::WebSocketHandshake::clientHandshake(co
  testWSClient.exe!ix::WebSocketTransport::connectToUrl(const
  testWSClient.exe!ix::WebSocket::connect(int timeoutSecs) Li
  testWSClient.exe!ix::WebSocket::checkConnection(bool firstC
  testWSClient.exe!ix::WebSocket::run() Line 367 C++

IXWebClient, how key/cert are set :
    ix::SocketTLSOptions tlsOptions;
    tlsOptions.certFile = "WebRTC.test.crt";
    tlsOptions.keyFile = "WebRTC.test.key";
    tlsOptions.caFile = "WebRTC-CA.pem";
    webSocket.setTLSOptions(tlsOptions);
    std::string url("wss://localhost:8080");
    webSocket.setUrl(url);

No matter if the path of the key/certificate exists or not, I have the same error message from OpenSSL, which is weird...

So :
1. Any idea why I have certificate verify failed?
2. Is it normal that s->verify_mode is equal to SSL_VERIFY_PEER and i equal to 0
3. Is it normal that SSL_OP_NO_TLSv1_3 is enabled in the code?

Thanks a lot for any help.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mta.openssl.org/pipermail/openssl-users/attachments/20221216/fcad1720/attachment-0001.htm>