[openssl-project] The problem of (implicit) relinking and changed behaviour

[Benjamin Kaduk]

On Sun, Apr 15, 2018 at 01:49:29PM +0200, Richard Levitte wrote:
> In message <ac42d34d-4ad9-3646-19a9-e9e3521d615a at openssl.org> on Sun, 15 Apr 2018 13:27:15 +0200, Andy Polyakov <appro at openssl.org> said:
> 
> appro> To summarize, failing tests in 110 should be revisited to see if they
> appro> are actually representative, before one can consider as drastic measures
> appro> as #5945.
> 
> At this point, I agree.  Viktor's look at several applications and
> Kurt's report of successful did convince me that I might by crying
> wolf a bit much.  I've started looking a bit deeper at the failing
> tests, for exactly the reasons you mention.

I'll echo the thanks going around for looking into the tests more
closely, and note that I'm happy to see that you are stepping back a
bit from the initial report -- it seems good be concerned, but I
don't think we have reason to panic quite yet.

Having said that, I'll step back a bit and try to speak to the
"philosophy" question from the start of the thread.  I think it is
good philosophy to worry about this sort of ABI stability, and agree
with Tim that it's good release engineering practice to have
automated tests for it (to the extent that we can).  What I'd like
us to think more about, though, is our release engineering in
general.  Don't get me wrong; we've made huge progess since the
1.0.2 era already, adding much more comprehensive test suite
(including external tests) and insisting that new features come with
tests and documentation.  But I still wonder if we could/should be
more aware of the release engineering consequences of our more daily
changes, at least when master is intended to be on a "stable ABI"
branch.

In the following examples, I'm not trying to point fingers or say
that we made mistakes; so far as I know these were all consistent
with our current policy and procedures.  That said, I have to wonder
if doing things like reverting old changes that we don't have CLAs
for, or introducing additional churn into the RNG, are the best idea
in this stage of the release cycle.  True, we're only in feature
freeze and not some final release freeze, but there is risk/reward
analysis to be done, and I just don't know how much we've been
thinking about that sort of tradeoff.  (I believe there are other
examples in a similar vein, but don't remember them off the top of
my head.)  I guess this can be summarized as "I don't have a good
understanding for what scale/size of change we as a group are
comfortable making during various stages of ABI compatibility and
release planning" -- I don't want to dictate policy, just start a
discussion (or at least gain a better understanding of existing
consensus).

I will note, though, that if we want to be very contemplative about
the release engineering and ABI stability consequences of changes
landing on an "ABI-stable" branch, that may end up leading to a
stronger argument for having master never be the "ABI-stable"
branch, so that it's always open for development work and we need
not do exhaustive analysis for *all* commits during some
months+-long period of time.  (There would then be a separate stable
branch, of course, and we'd do the extra "thinking" when merging to
it.)

-Ben