How to fix "OpenSSL failed - error:0A000086:SSL routines::certificate verify failed"

[psv sridhar]

you are sending flooded emails wrongly. stop it.
 Thanks and Regards
Sridhar PSVPhone 571 244-5862 

    On Friday, December 16, 2022 at 04:08:38 PM CST, Pierre-Luc Boily <pierreluc.boily at gmail.com> wrote:  
 
 Hello,

Details
OS : WIndows 10Arch : x64Compiler : VisualStudio 2017
I have a c++ wss IXWebSocket client that tries to connect to a nodejs https/websocket server but the client refuses to connect and returns the error : OpenSSL failed - error:0A000086:SSL routines::certificate verify failed
What I tried
   
   - I have a React front end using wss to communicate to my https nodejs server. It works -> This confirms that my key and certificate are valid.
   - I also tried the same c++ client above, not secured (no wss) connecting to my same nodejs server, but http/websocket (non secure). It works.
So, I had to dig into the OpenSSL code and I found where the error is triggered, see code below.  In my case s->verify_mode is equal to SSL_VERIFY_PEER and i equal to 0 and I don't know if those values are OK or not.
While I was digging into the code, I also realized that SSL_OP_NO_TLSv1_3 is automagically defined for my code.  I feel that it is incorrect.
>From statem_clnt.c line 1888:    if (s->verify_mode != SSL_VERIFY_NONE && i <= 0) {
        SSLfatal(s, ssl_x509err2alert(s->verify_result),
                 SSL_R_CERTIFICATE_VERIFY_FAILED);
        return WORK_ERROR;
    }

Stacktrace:> libssl-3-x64.dll!tls_post_process_server_certificate(ssl_st
  libssl-3-x64.dll!ossl_statem_client_post_process_message(ss
  libssl-3-x64.dll!read_state_machine(ssl_st * s) Line 675 
  libssl-3-x64.dll!state_machine(ssl_st * s, int server) Line
  libssl-3-x64.dll!ossl_statem_connect(ssl_st * s) Line 266 
  libssl-3-x64.dll!SSL_do_handshake(ssl_st * s) Line 3937 C  
  libssl-3-x64.dll!SSL_connect(ssl_st * s) Line 1760 C      
  testWSClient.exe!ix::SocketOpenSSL::openSSLClientHandshake(
  testWSClient.exe!ix::SocketOpenSSL::connect(const std::basi
  testWSClient.exe!ix::WebSocketHandshake::clientHandshake(co
  testWSClient.exe!ix::WebSocketTransport::connectToUrl(const
  testWSClient.exe!ix::WebSocket::connect(int timeoutSecs) Li
  testWSClient.exe!ix::WebSocket::checkConnection(bool firstC
  testWSClient.exe!ix::WebSocket::run() Line 367 C++        

IXWebClient, how key/cert are set :    ix::SocketTLSOptions tlsOptions;
    tlsOptions.certFile = "WebRTC.test.crt";
    tlsOptions.keyFile = "WebRTC.test.key";
    tlsOptions.caFile = "WebRTC-CA.pem";
    webSocket.setTLSOptions(tlsOptions);
    std::string url("wss://localhost:8080");
    webSocket.setUrl(url);

No matter if the path of the key/certificate exists or not, I have the same error message from OpenSSL, which is weird...
So : 1. Any idea why I have certificate verify failed?2. Is it normal that s->verify_mode is equal to SSL_VERIFY_PEER and i equal to 03. Is it normal that SSL_OP_NO_TLSv1_3 is enabled in the code?
Thanks a lot for any help.  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mta.openssl.org/pipermail/openssl-users/attachments/20221216/6f785cda/attachment.htm>