[openssl-dev] State machine rewrite

[Matt Caswell]

I've just opened a github pull request to show recent work I have been
doing on rewriting the OpenSSL state machine (for version 1.1.0). See:
https://github.com/openssl/openssl/pull/394

My objectives for the rewrite were:

- Remove duplication of state code between client and server
- Remove duplication of state code between TLS and DTLS
- Simplify transitions and bring the logic together in a single location
so that it is easier to validate
- Remove duplication of code between each of the message handling functions
- Receive a message first and then work out whether that is a valid
transition - not the other way around (the other way causes lots of
issues where we are expecting one type of message next but actually get
something else)
- Separate message flow state from handshake state (in order to better
understand each)
  - message flow state = when to flush buffers; handling restarts in the
event of NBIO events; handling the common flow of steps for reading a
message and the common flow of steps for writing a message etc
  - handshake state = what handshake message are we working on now
- Control complexity: only the state machine can change state: keep all
the state changes local to the state machine component

The message flow state machine is divided into a reading sub-state
machine and a writing sub-state machine. See the source comments in
ssl/statem/statem.c for a more detailed description of the various
states and transitions possible. Also see ssl/statem/README for
additional info.

One issue is that the patch as it is currently removes support for
DTLSv1_listen. I have another patch to add that back in (in a completely
different way) - but it needs a bit more work yet.

I am interested in hearing any feedback you may have on the code
(ideally as comments in the pull request). I would also be keen to hear
of any problems you might encounter whilst using this code. You can
check it out from my github repo:
https://github.com/mattcaswell/openssl

See the state-machine-rewrite branch.

Thanks

Matt