[openssl-project] 1.1.1 Release timetable

[Matt Caswell]

The recent f2f meeting gave me the action of pulling together a release
timetable for 1.1.1.

For the TL;DR summary skip to "1.1.1 Release timetable proposal below".

Background
==========

I looked back at what we did for 1.1.0. This is the vote that we passed
for our original (intended) timetable:

The team will adopt the following target dates for the 1.1.0 release.
This timetable will be published in the release strategy as
"aspirational" (or similar words).

10th December 2015, alpha release 1
7th January 2016, alpha release 2
4th February 2016, alpha release 3
3rd March 2016, 1.1.0 beta 1 release,
	OpenSSL_1_1_0-stable branch created
	master becomes 1.1.1 basis
31st March 2016, 1.1.0 beta 2 release
28th April 2016, 1.1.0 public release

An alpha release means:
- Not (necessarily) feature complete
- Not necessarily all new APIs in place yet
- Opaque work complete

A beta release means:
- Feature complete/Feature freeze
- Bug fixes only


You will notice that those dates are each 4 weeks apart. The reality of
what actually happened was somewhat different. We actually achieved this:

10th December 2015, alpha release 1
14th January 2016, alpha release 2
15th February 2016, alpha release 3
16th March 2016, 1.1.0 beta 1 release
19th April 2016, 1.1.0 beta 2 release
4th August 2016, 1.1.0 beta 3 release
25th August 2016, 1.1.0 public release

[Aside: Interestingly, from the "old" link of the download page, I only
see pre1, pre2 and pre3 available for download. Anyone know what
happened to pre4, pre5 and pre6 downloads?]

So although we slipped a bit we more or less stuck to the plan until
beta2. In April 2016 we were still planning to do a final release after
just 2 betas and were targeting 12th May 2016. In early May we started
to worry that despite the alpha/beta process we were not ready for a
release. Most notably we were concerned about the number of open defects
which had been hanging around for a long time. After much debate we
defined some release criteria (these were never actually voted on, but
this seems to have been the consensus view). The release criteria were:

- No open RT tickets more than five years old
- No open RT tickets or GitHub PRs/Issues flagged with the 1.1.0
milestone and older than 2 weeks at the time of release (all new tickets
prior to release to be assessed and 1.1.0 milestone added if they are
relevant to that branch)
- Clean builds in Travis, Appveyor and Cisco for two days
- run-checker.sh to be showing as clean 2 days before release
- No open Coverity issues (not flagged as "False Positive" or "Ignore").

The very late definition of these criteria set us back significantly and
we then spent the next few months trying to achieve those criteria.
Eventually we decided that we were nearly there, but a lot of time had
passed so we decided to add a third beta release. From there things went
more or less to plan and we eventually released on 25th August 2016.

One other thing that (I think) didn't help was that our first beta
release was really more of an alpha. We did not do a feature freeze
until beta1, so new stuff was going in all the way up to that release.
In my mind a beta release implies some level of stability which we just
didn't have at that point in time. Probably we should have called the
feature freeze at alpha3 to give things time to settle down before beta1.

There are a few different circumstances for 1.1.1 which we need to keep
in mind for our release plan. Most importantly we don't know when
TLSv1.3 will be finalised. We hope that it won't be long, but we just
don't know. The other difference this time around (I think) is that more
people seem to be testing 1.1.1 than was the case for 1.1.0. At least
that is my perception based on the number of
questions/posts/issues/patches etc that we are getting for it.


1.1.1 Release timetable proposal
================================

With all of the above in mind here is my straw man proposal for 1.1.1:

23rd January 2018, alpha release 1 (pre1)
20th February 2018, alpha release 2 (pre2)
	OpenSSL_1_1_1-stable created (feature freeze)
	master becomes basis for 1.1.2 or 1.2.0 (TBD)
20th March 2018, beta release 1 (pre3)
17th April 2018, 1.1.1 public release (assuming TLSv1.3 RFC is published)

An alpha release means:
- Not (necessarily) feature complete
- Not necessarily all new APIs in place yet

A beta release means:
- Feature complete/Feature freeze
- Bug fixes only

Note that alpha release 2 may have new features in it, but we will
freeze at that point so that beta release 1 will not have new features.

My proposed release criteria are:
- All open github issues/PRs older than 2 weeks at the time of release
to be assessed for relevance to 1.1.1. Any flagged with the 1.1.1
milestone to be closed (see below)
- Clean builds in Travis and Appveyor for two days
- run-checker.sh to be showing as clean 2 days before release
- No open Coverity issues (not flagged as "False Positive" or "Ignore")
- TLSv1.3 RFC published

Valid reasons for closing an issue/PR with a 1.1.1 milestone might be:
- We have just now or sometime in the past fixed the issue
- Unable to reproduce (following discussion with original reporter if
possible)
- Working as intended
- Deliberate decision not to fix until a later release
- Not enough information and unable to contact reporter
- etc

With this plan my expectation is that from 20th February (feature
freeze) the focus will shift from features to closing off open
issues/PRs. By doing so much earlier than with 1.1.0 I hope to avoid the
drawn out release timetable.

If the TLSv1.3 RFC is not published by the time we are ready to release,
or we haven't made the progress we want on the other release criteria
then we can add additional betas as we see fit until such time as we are
ready.

Matt