[openssl-users] Multiplexing TLS / non-TLS connections on a single socket

[Jakob Bohm]

On 12/02/2019 21:23, Paul Smith wrote:
> Hi all.
>
> We have a service that currently implements a home-grown secure
> connection model based on SRP using AES as the cipher.  We want to add
> support for TLS 1.2/1.3 as well, but we have to maintain backward-
> compatibility.  Our app is in C++ and using OpenSSL 1.1.1.
>
> We really don't want to create a separate socket: we'd like to support
> client requests on the same socket using either the old connection
> method or TLS.  We also want to support "pure" TLS, rather than some
> kind of wrapped connection protocol.  This means we need to determine
> at connect time which method is being used.
>
> One idea is to use MSG_PEEK on the socket recv() to check the first
> bytes of the initial message (our protocol uses an XML message as the
> initial connection so seeing something like "<Connect" would be enough
> to differentiate them).  One possible annoyance is that we need to
> support Windows as well as GNU/Linux and I understand that peek on
> Winsocket is not very efficient.
>
> Is PEEK still the best bet?  Or is there a way in OpenSSL to manage
> this more directly?  For example we read the initial message then if we
> discover that it's not the old connection model we provide it plus the
> socket to OpenSSL so it can handle the rest of the handshake?  Or maybe
> we can register a callback with OpenSSL so that if it reads an initial
> message from the socket that it doesn't recognize it would hand that
> back to us?
>
> Any pointers to docs and/or examples would be really helpful, thanks!
>
At least in older versions of OpenSSL, you could create a custom BIO
that buffers the socket data and lets you look at it before passing
it to the SSL/TLS layer or directly to your code according to the
contents.  This way you don't depend on the ability to make the OS
socket API do this for you.

I don't know if this ability is also in OpenSSL 1.1.x.

Enjoy

Jakob
-- 
Jakob Bohm, CIO, Partner, WiseMo A/S.  https://www.wisemo.com
Transformervej 29, 2860 Søborg, Denmark.  Direct +45 31 13 16 10
This public discussion message is non-binding and may contain errors.
WiseMo - Remote Service Management for PCs, Phones and Embedded