IXWebSocket wss c++ client cannot connect to Node.js wss server using an ip address

[Viktor Dukhovni]

On Tue, Feb 14, 2023 at 11:50:25AM -0500, Pierre-Luc Boily wrote:
> Thanks a lot for this information.  I was also just browsing and debugging
> this exact file, it might not do any harm to understand a little bit more
> how OpenSSL works.....  My traces show that the problem is not coming from
> the function  you are pointing to, but from line 529 :
> 
> SSL_CTX_set_verify(_ssl_context,
> SSL_VERIFY_PEER,
> [](int preverify, X509_STORE_CTX*) -> int { return preverify; });

That callback lambda is not needed, a NULL pointer would have worked
just as well, but sure, if the library also calls:

    SSL_set1_host(_ssl, "<ipaddress>");

then OpenSSL will perform an internal hostname check, which will fail,
because the IP address in the certificate is not a hostname.

> From my understanding, this function is verifying the certificate on a
> callback.

No, that function is just a NOP, that needlessly intercepts and parrots
the already detected failure.

> "[](int preverify, X509_STORE_CTX*) -> int { return preverify; })"
> 
> returns 0, which means it failed.

No, "it" did not fail, it merely parrots the failure.

> That is not really clear to me why, and
> what does X509_STORE_CTX .  I guess that prior to the SSL_CTX_set_verify, I
> have to do something differently?

The right answer is: don't use this library.

If you must use the library, then when connecting to an IP address:

    /* clear the hostname */
    SSL_set1_host(_ssl, NULL);

    /* Configure the IP address */
    X509_VERIFY_PARAM *param = SSL_get0_param(_ssl);
    X509_VERIFY_PARAM_set1_ip_asc(param, "<ipaddress>");

when connecting to a hostname:


    /* Set the hostname */
    SSL_set1_host(_ssl, "<hostname>");

> Like calling SSL_set1_host somewhere

No, because the IP address is not a hostname.

-- 
    Viktor.