How to fix "OpenSSL failed - error:0A000086:SSL routines::certificate verify failed"

[Pierre-Luc Boily]

Hello,

*Details*
OS : WIndows 10
Arch : x64
Compiler : VisualStudio 2017

I have a *c++ wss IXWebSocket
<https://github.com/machinezone/IXWebSocket> client* that tries to connect
to a *nodejs https/websocket server* but the client refuses to connect and
returns the error : *OpenSSL failed - error:0A000086:SSL
routines::certificate verify failed*
*What I tried*

   1. I have a React front end using wss to communicate to my https nodejs
   server. *It works ->* *This confirms that my key and certificate are
   valid.*
   2. I also tried the same c++ client above, not secured (no wss)
   connecting to my same nodejs server, but http/websocket (non secure). *It
   works*.

So, I had to dig into the OpenSSL code and I found where the error is
triggered, see code below.  In my case *s->verify_mode* is equal to
*SSL_VERIFY_PEER* and *i* equal to *0* and I don't know if those values are
OK or not.

While I was digging into the code, I also realized that *SSL_OP_NO_TLSv1_3*
is automagically defined for my code.  I feel that it is incorrect.

*From statem_clnt.c line 1888*:
    if (s->verify_mode != SSL_VERIFY_NONE && i <= 0) {
        SSLfatal(s, ssl_x509err2alert(s->verify_result),
                 SSL_R_CERTIFICATE_VERIFY_FAILED);
        return WORK_ERROR;
    }

*Stacktrace*:
> libssl-3-x64.dll!tls_post_process_server_certificate(ssl_st
  libssl-3-x64.dll!ossl_statem_client_post_process_message(ss
  libssl-3-x64.dll!read_state_machine(ssl_st * s) Line 675
  libssl-3-x64.dll!state_machine(ssl_st * s, int server) Line
  libssl-3-x64.dll!ossl_statem_connect(ssl_st * s) Line 266
  libssl-3-x64.dll!SSL_do_handshake(ssl_st * s) Line 3937 C
  libssl-3-x64.dll!SSL_connect(ssl_st * s) Line 1760 C
  testWSClient.exe!ix::SocketOpenSSL::openSSLClientHandshake(
  testWSClient.exe!ix::SocketOpenSSL::connect(const std::basi
  testWSClient.exe!ix::WebSocketHandshake::clientHandshake(co
  testWSClient.exe!ix::WebSocketTransport::connectToUrl(const
  testWSClient.exe!ix::WebSocket::connect(int timeoutSecs) Li
  testWSClient.exe!ix::WebSocket::checkConnection(bool firstC
  testWSClient.exe!ix::WebSocket::run() Line 367 C++

*IXWebClient, how key/cert are set :*
    ix::SocketTLSOptions tlsOptions;
    tlsOptions.certFile = "WebRTC.test.crt";
    tlsOptions.keyFile = "WebRTC.test.key";
    tlsOptions.caFile = "WebRTC-CA.pem";
    webSocket.setTLSOptions(tlsOptions);
    std::string url("wss://localhost:8080");
    webSocket.setUrl(url);

No matter if the path of the key/certificate exists or not, I have the same
error message from OpenSSL, which is weird...

*So :*
1. Any idea why I have *certificate verify failed*?
2. Is it normal that *s->verify_mode* is equal to *SSL_VERIFY_PEER*
and *i* equal
to *0*
3. Is it normal that *SSL_OP_NO_TLSv1_3* is enabled in the code?

Thanks a lot for any help.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mta.openssl.org/pipermail/openssl-users/attachments/20221216/11778b0f/attachment-0001.htm>