[openssl-project] Is making tests faster a bugfix?

[Tim Hudson]

Improved testing to me is something that is a good thing - and a value
judgement.
It doesn't change libcrypto or libssl - and that to me is the way I think
of it.
Fixing tests and apps and Makefiles to me are different from adding
features to libcrypto or libssl.

On this one - the fuzz testing has been sufficiently slow to reduce its
usefulness - and this is a step in the right direction.

It is however also a bit outside of our current policy on such things - so
perhaps we need to update that.

Tim.


On Thu, Mar 29, 2018 at 11:45 PM, Richard Levitte <levitte at openssl.org>
wrote:

> In message <e32db73b-b91d-d3c2-274b-6e13e969af34 at openssl.org> on Thu, 29
> Mar 2018 14:03:06 +0100, Matt Caswell <matt at openssl.org> said:
>
> matt>
> matt>
> matt> On 29/03/18 14:00, Salz, Rich wrote:
> matt> > Please see https://github.com/openssl/openssl/pull/5788
> matt> >
> matt> > I don’t think it is, but I’d like to know what others think.
> matt>
> matt> I do think this should be applied. The tests in question are not just
> matt> slow but *really* slow to the point that I often exit them before
> they
> matt> have completed. This removes the benefits of having the tests in the
> matt> first place. From that perspective I view this as a bug fix.
>
> Something to remember is that no user will ever complain about this,
> because we don't deliver the contents of fuzz/corpora in our tarballs.
>
> In other words, this is a developer only change of our current tests,
> and you will only hear from developers who do engage in fuzz testing,
> i.e. those who do these tests as part of a release, just to pick a
> very recent example.
>
> Also, you may note that this test re-engages fuzz testing as part of
> our normal tests that are run for every PR, which means that we will
> catch errors that the fuzzers can detect much earlier.  Because the
> fuzz testing took so long time, we had them only engaged with
> [extended tests], something that's almost never used.
>
> So I would argue that faster fuzz testing means more fuzz testing, and
> hopefully better testing of stuff that's harder to catch otherwise.
>
> Cheers,
> Richard ( plus, from a very personal point of view, it's *my* time,
>           and Matt's, and whoever else's who tests for releases, that
>           gets substantially less wasted! )
>
> --
> Richard Levitte         levitte at openssl.org
> OpenSSL Project         http://www.openssl.org/~levitte/
> _______________________________________________
> openssl-project mailing list
> openssl-project at openssl.org
> https://mta.openssl.org/mailman/listinfo/openssl-project
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mta.openssl.org/pipermail/openssl-project/attachments/20180329/d91f5cd3/attachment-0001.html>