IXWebSocket wss c++ client cannot connect to Node.js wss server using an ip address

[Michael Wojcik]

From: openssl-users <openssl-users-bounces at openssl.org> On Behalf Of Pierre-Luc Boily
Sent: Monday, 13 February, 2023 13:08

[Mark Hack wrote]

>> Looking at the IX code (and it was a very quick look), I suspect that only the CN is validated.

> Indeed, server cert shows "localhost", but it also shows the ip address :
> 
>             X509v3 Subject Alternative Name:
>                 DNS:localhost, IP Address:192.168.230.138, IP Address:127.0.0.1

Yes, but those are SANs. If Mark is correct and the library only uses the Subject CN when performing the name-checking step of verification, then the SANs won't help.

> By the way, the author of the IXWebSocket c++ library told me :
> 
> This might be a high level SSL stuff, where you actually need a real hostname and can't use an IP.

Supporting SANs is hardly "high level" -- more like "bare minimum for correct functioning in the modern TLS environment".

You can most certainly use IPADR SANs when verifying peer certificates (we do that routinely in testing, since some customers have uses for them). That's why they exist.

> But on the other hand, I have a javascript websocket client  that can connect to my wss server
> using the same certificate as the c++ client.

Yes, this too suggests that the IXWebSocket implementation does not do certificate verification properly.

>  But as a workaround, I thought to use a fake domain name that points to the server IP address. 
> Maybe this will work?

Since you're apparently up against a deficiency in your client library, it's hard for us to guess what will work. That said, yes, you can create a certificate with any names (where, in the case of normal TLS server-certificate verification, "name" could be a FQDN, a bare hostname, or an IP address) as SANs; and you can, if you wish, use one of those names as the CN component of the Subject DN, to cater to clients which don't to verification correctly. (The CA/BF Basic Requirements mandate ignoring the Subject DN if SANs are present, for purposes of verifying the certificate identity.)

For verification, the rules for matching the name(s) present in the server's certificate with the name intended by the client are complicated, thanks to SANs, wildcarding, and so on. There are also API issues: prior to OpenSSL 1.1.0, OpenSSL didn't do it for you (so name matching had to be done by the application); with 1.1.0 and later, you have to tell OpenSSL to do the name matching and what name you want it to match. A great many clients still get this wrong. (This is very common with mobile applications, for example.)

Generally speaking, if the client does it correctly, then the server's certificate should contain a SAN that matches (case-insensitively, and possibly using wildcards, though not a wildcard followed by a forbidden domain) the exact name intended by the client. Of course the client has to tell OpenSSL what name it intends to find in the server's certificate. Then all the other verification checks have to pass (certificate signatures are good, OpenSSL can chain back to a trusted root, nothing's expired, Basic Constraints and EKUs are correct, maybe you're a masochist and you enabled CRLs and/or OSPF, blah blah blah).

As usual, a good rule of thumb is that TLS is hard, PKIX (Public Key Infrastructure - X.509) is worse, and most developers get them wrong in some number of ways, regardless of what library they're using. I've been working with them for a quarter of a century; I've read books and I've read the specs; I've dealt with hundreds of real-world use case; and I still keep running into things I hadn't seen before, or having to dig through references to make sure I'm remembering something correctly.

-- 
Michael Wojcik