[openssl-project] [TLS] Yet more TLS 1.3 deployment updates

[Sam Roberts]

I appreciate that there are multiple ways of looking at this, but I
don't think that looking at the info callback from the point of view
of "is it or is it not a handshake record" is pragmatic. The
SSL_set_msg_callback could be used by people wanting to see all the
underlying records.

The way the info callbacks are behaving for TLS1.3 is a major obstacle
to Node.js supporting TLS1.3. The intention of 1.1.1 was to make
TLS1.3 support as painfree as possible, and it doesn't seem to be
doing that. When it was just Node.js that was having to work around
this, I was convinced that we had some kind of historical abuse of the
info callback we would just have to fix. Now that I hear that (at
least) haproxy and nginx are also affected, I'd appreciate a
reconsideration of this from the purely pragmatic point of view of
"how do we make TLS1.3 support easy to achieve", the ultimate goal
here for everyone (I think!).

The behaviour of the info callbacks is already backwards incompatible
with openssl 1.1.0, even for TLS1.2
(https://github.com/openssl/openssl/issues/7199). For API users
wanting to detect ticket exchange, HelloRequest, and other "handshake"
records with the info callbacks, changing to use different "where"
values would not be as disruptive as the affect is on users trying to
detect handshake start/done via callback. If I am underestimating how
disruptive that is, its not out of the question to have a
"compatibility" mode where the info callbacks change their behaviour
based on how the context is set up No one likes having to support
these kinds of flags, but it would remove concerns about introducing
breaking changes in the 1.1.1 line if they were opt-in, so this
doesn't become a question of "who's code do we break".

Its also not clear whether all uses of the info callback actually have
another API alternatives.

We use the info callbacks to reject renegotiation. Apparently there is
an option to do this now with a mode, so we could likely adopt that
way.

We use the info callbacks to time renegotiations, and reject them if
they take too long. I'm not aware of any other API that would allow us
to do this.

We  use the info callbacks to detect when renegotiation has been
requested on an existing connection, and inform node API users. I'm
not aware of any other API that would allow us to  do that.

Changing from using the info callbacks to get notified of start/done
of the initial handshake using SSL_do_handshake() (or
SSL_connect/SSL_accept) is theoretically possible, but turns out to be
easy to say, and has been much harder to do. The interactions with
node.js internal stream and async mechanisms are subtle, and after
close to a week of concerted effort, I still haven't gotten it working
(some things work, some don't, its been a discouraging game of
whack-a-bug). It can't be impossible, but its the type of substantial
rewrite necessary to support TLS1.3 that ideally would not be
necessary.

The other changes that TLS1.3 requires, multiple session tickets, a
few new APIs to replace some of the SSL_renegotiate use-cases, etc.,
all are pretty routine. We could get TLS1.3 support in Node.js fairly
quickly if the info callback issue was solved openssl side. I'm even
happy to help work on it if that's an issue, it would be more
productive than what I've been trying to do in Node.js.

Thanks,
Sam