[openssl-dev] Running against BoringSSL's SSL test suite

[David Benjamin]

Hi folks,

So, we've by now built up a decent-sized SSL test suite in BoringSSL. I was
bored and ran it against OpenSSL master. It revealed a number of bugs. One
is https://github.com/openssl/openssl/pull/603. I'll be filing tickets
shortly for the remaining ones I've triaged, but I thought I'd send this
separately rather than duplicate it everywhere.

Emilia also suggested there may be room to collaborate on testing. If
nothing else, just borrowing ideas or porting tests to/from your TLSProxy
setup. (Like, say, the ones that caught the bugs I'll be reporting. :-) )
So, here's an introduction on how it all works:

To run the tests on OpenSSL, clone BoringSSL:
https://boringssl.googlesource.com/boringssl/
Then patch in this change. (Click the "Download" in the upper-right for
options.)
https://boringssl-review.googlesource.com/#/c/7332/
Then follow the instructions in the commit message.

The tests themselves and the runner logic live in ssl/test/runner/runner.go:
https://boringssl.googlesource.com/boringssl/+/22ce9b2d08a52e399bf2ab86851952d727be034d/ssl/test/runner/runner.go#922

They work by running an unmodified TLS stack in a shim binary against a
copy of Go's. We patch our copy with options for weird behavior to test
against:
https://boringssl.googlesource.com/boringssl/+/22ce9b2d08a52e399bf2ab86851952d727be034d/ssl/test/runner/common.go#414

Go and shim communicate entirely with sockets and (tons of) command-line
flags, though it is slightly overfit to BoringSSL's behavior and checks
error strings a lot. The shim also has options like -async mode which we
use on a subset of tests to stress state machine resumption. (This has
saved me from state machine bugs so many times.)
https://boringssl.googlesource.com/boringssl/+/22ce9b2d08a52e399bf2ab86851952d727be034d/ssl/test/runner/runner.go#2770
https://boringssl.googlesource.com/boringssl/+/22ce9b2d08a52e399bf2ab86851952d727be034d/ssl/test/bssl_shim.cc#826

I hope this is useful! Bugs and patches will follow this mail, as I write
them up.

David
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mta.openssl.org/pipermail/openssl-dev/attachments/20160307/95675d97/attachment-0001.html>