[openssl-dev] Running against BoringSSL's SSL test suite

Matt Caswell matt at openssl.org
Tue Mar 8 10:10:17 UTC 2016

On 07/03/16 21:49, David Benjamin wrote:
> Hi folks,
> So, we've by now built up a decent-sized SSL test suite in BoringSSL. I
> was bored and ran it against OpenSSL master. It revealed a number of
> bugs. One is https://github.com/openssl/openssl/pull/603. I'll be filing
> tickets shortly for the remaining ones I've triaged, but I thought I'd
> send this separately rather than duplicate it everywhere.

Wow! That's awesome! Thanks David.

> Emilia also suggested there may be room to collaborate on testing. If
> nothing else, just borrowing ideas or porting tests to/from your
> TLSProxy setup. (Like, say, the ones that caught the bugs I'll be
> reporting. :-) ) So, here's an introduction on how it all works:
> To run the tests on OpenSSL, clone BoringSSL:
> https://boringssl.googlesource.com/boringssl/
> Then patch in this change. (Click the "Download" in the upper-right for
> options.)
> https://boringssl-review.googlesource.com/#/c/7332/
> Then follow the instructions in the commit message.
> The tests themselves and the runner logic live in ssl/test/runner/runner.go:
> https://boringssl.googlesource.com/boringssl/+/22ce9b2d08a52e399bf2ab86851952d727be034d/ssl/test/runner/runner.go#922
> They work by running an unmodified TLS stack in a shim binary against a
> copy of Go's. We patch our copy with options for weird behavior to test
> against:
> https://boringssl.googlesource.com/boringssl/+/22ce9b2d08a52e399bf2ab86851952d727be034d/ssl/test/runner/common.go#414
> Go and shim communicate entirely with sockets and (tons of) command-line
> flags, though it is slightly overfit to BoringSSL's behavior and checks
> error strings a lot. The shim also has options like -async mode which we
> use on a subset of tests to stress state machine resumption. (This has
> saved me from state machine bugs so many times.)
> https://boringssl.googlesource.com/boringssl/+/22ce9b2d08a52e399bf2ab86851952d727be034d/ssl/test/runner/runner.go#2770
> https://boringssl.googlesource.com/boringssl/+/22ce9b2d08a52e399bf2ab86851952d727be034d/ssl/test/bssl_shim.cc#826
> I hope this is useful! Bugs and patches will follow this mail, as I
> write them up.

Great. We're in the final few days prior to the 1.1.0 feature freeze and
the team are working flat out at the moment. I'll try and start looking
at them once we're past that milestone later this week.


More information about the openssl-dev mailing list