[openssl-project] Is making tests faster a bugfix?

[Richard Levitte]

In message <e32db73b-b91d-d3c2-274b-6e13e969af34 at openssl.org> on Thu, 29 Mar 2018 14:03:06 +0100, Matt Caswell <matt at openssl.org> said:

matt> 
matt> 
matt> On 29/03/18 14:00, Salz, Rich wrote:
matt> > Please see https://github.com/openssl/openssl/pull/5788
matt> > 
matt> > I don’t think it is, but I’d like to know what others think.
matt> 
matt> I do think this should be applied. The tests in question are not just
matt> slow but *really* slow to the point that I often exit them before they
matt> have completed. This removes the benefits of having the tests in the
matt> first place. From that perspective I view this as a bug fix.

Something to remember is that no user will ever complain about this,
because we don't deliver the contents of fuzz/corpora in our tarballs.

In other words, this is a developer only change of our current tests,
and you will only hear from developers who do engage in fuzz testing,
i.e. those who do these tests as part of a release, just to pick a
very recent example.

Also, you may note that this test re-engages fuzz testing as part of
our normal tests that are run for every PR, which means that we will
catch errors that the fuzzers can detect much earlier.  Because the
fuzz testing took so long time, we had them only engaged with
[extended tests], something that's almost never used.

So I would argue that faster fuzz testing means more fuzz testing, and
hopefully better testing of stuff that's harder to catch otherwise.

Cheers,
Richard ( plus, from a very personal point of view, it's *my* time,
	  and Matt's, and whoever else's who tests for releases, that
	  gets substantially less wasted! )

-- 
Richard Levitte         levitte at openssl.org
OpenSSL Project         http://www.openssl.org/~levitte/