[openssl-project] Speeding up the fuzz test...

[Richard Levitte]

Hey,

Matt and I talked a bit about the fuzz test recipe and how
*incredibly* slow it is, especially on all things Windows.  We
realised fairly quickly that this is caused by a gazillion program
invokations (once for every corpora file, that's more than 15000 of
them), and then we started thinking about archiving the corpora
somehow.

After having a look through diverse format, I kinda fell for the cpio
formats.  They're very simple, putting together a reader was fairly
easy, and well, the result of running a modified test_fuzz.t that uses
the cpio files instead of the corpora files directly was quite a
dramatic change!

Running with the raw corpora files:

    : ; make test TESTS=test_fuzz
    ...
    All tests successful.
    Files=1, Tests=11, 457 wallclock secs ( 3.54 usr  0.28 sys + 330.48 cusr 189.06 csys = 523.36 CPU)
    Result: PASS

And the cpio variant:

    : ; make test TESTS=test_fuzz_new
    ...
    All tests successful.
    Files=1, Tests=11, 35 wallclock secs ( 0.01 usr  0.00 sys + 35.17 cusr  0.11 csys = 35.29 CPU)
    Result: PASS

And this was a Linux, which is substantially faster than my Windows 10
box, where I can wait for quite a while (we're talking half an hour,
maybe more).  I just wonder what kind of results I'll see there.

Now, I wonder how that will impact on Kurt, who sometimes produce
these files, and on Google's oss-fuzz project, who do use this.
My desire is to replace the current corpora with the corresponding
cpio files, one for each test program (i.e. fuzz/corpora/asn1/* gets
archived into fuzz/corpora/asn1.cpio, and so on and so forth).

I've changed fuzz/test-corpus.c so it can take a flag '-cpio' to tell
it to read the files as cpio archives, otherwise it read the file raw,
as before.

Thoughts?  Comments?  (Kurt?)

Cheers,
Richard

-- 
Richard Levitte         levitte at openssl.org
OpenSSL Project         http://www.openssl.org/~levitte/