[openssl-users] Multiplexing TLS / non-TLS connections on a single socket

[Paul Smith]

Hi all.

We have a service that currently implements a home-grown secure
connection model based on SRP using AES as the cipher.  We want to add
support for TLS 1.2/1.3 as well, but we have to maintain backward-
compatibility.  Our app is in C++ and using OpenSSL 1.1.1.

We really don't want to create a separate socket: we'd like to support
client requests on the same socket using either the old connection
method or TLS.  We also want to support "pure" TLS, rather than some
kind of wrapped connection protocol.  This means we need to determine
at connect time which method is being used.

One idea is to use MSG_PEEK on the socket recv() to check the first
bytes of the initial message (our protocol uses an XML message as the
initial connection so seeing something like "<Connect" would be enough
to differentiate them).  One possible annoyance is that we need to
support Windows as well as GNU/Linux and I understand that peek on
Winsocket is not very efficient.

Is PEEK still the best bet?  Or is there a way in OpenSSL to manage
this more directly?  For example we read the initial message then if we
discover that it's not the old connection model we provide it plus the
socket to OpenSSL so it can handle the rest of the handshake?  Or maybe
we can register a callback with OpenSSL so that if it reads an initial
message from the socket that it doesn't recognize it would hand that
back to us?

Any pointers to docs and/or examples would be really helpful, thanks!