[openssl-dev] Running against BoringSSL's SSL test suite

Fri Mar 18 10:57:41 UTC 2016

FYI for easier use, this patch now lives at
https://github.com/google/openssl-tests, rebased against BoringSSL latest
(thanks David!) and OpenSSL-1.1.0-pre4 (Beta 1). I've also checked in a log
from Beta 1.

Cheers,
Emilia

On Thu, Mar 10, 2016 at 4:33 PM David Benjamin <davidben at google.com> wrote:

> On Thu, Mar 10, 2016 at 1:30 AM Kanaka Kotamarthy <kotamarthyd at gmail.com>
> wrote:
>
>> And also Openssl fails with Resume-Client-NoResume cases. Do you have
>> any report on which test cases do fail and reasons for the failure?
>>
>>
>> RT tickets 4387 through 4395 were the failures I've triaged. I'm sure
>> there's more things in there to look through.
>>
>> I don't believe Resume-Client-NoResume fails for me. Perhaps something
>> was fixed between master and 1.1.0-pre2.
>>
>>
>> Openssl doesn't gives any error. For Resume-Client-NoResume-SSL3-TLS11
>> test case, we expect the new session's handshake to be done with TLS11. But
>> with Openssl handshake is done using SSL3. As in ssl3_clear, we set back
>> s->version to s->method->version.
>>
>
> Oh, sorry, I keep forgetting our runner doesn't make it clear when a -test
> option fails to match anything. (I should fix that...) I looked
> for Resume-Client-NoResume without noticing it had suffixes. :-)
>
> I would expect most things addResumptionVersionTests to fail. See
> https://github.com/openssl/openssl/pull/603
>
> David
>
>
>> Thank you
>> Durga.
>>
>> On Wed, Mar 9, 2016 at 10:38 PM, David Benjamin <davidben at google.com>
>> wrote:
>>
>>> On Wed, Mar 9, 2016 at 5:07 AM Kanaka Kotamarthy <kotamarthyd at gmail.com>
>>> wrote:
>>>
>>>> Hi
>>>>
>>>> I am even testing OpenSSL with BoringSSL's test cases using
>>>> Openssl-1.1.0-pre2. Trying to find out reasons of OpenSSL's failures
>>>> for particular cases.
>>>>
>>>> DTLS 1.0 session resumption has some thing wrong. If s_server started
>>>> with -dtls and s_client -dtls1 -reconnect , session resumption is not
>>>> being done. The reason for this may be, version negotiation for DTLS
>>>> is done after loading previous session and check for s->version and
>>>> s->session->version fails in tls_process_client_hello.
>>>>
>>>
>>> See RT #4392.
>>> https://rt.openssl.org/Ticket/Display.html?id=4392
>>>
>>>
>>>> And also Openssl fails with Resume-Client-NoResume cases. Do you have
>>>> any report on which test cases do fail and reasons for the failure?
>>>>
>>>
>>> RT tickets 4387 through 4395 were the failures I've triaged. I'm sure
>>> there's more things in there to look through.
>>>
>>> I don't believe Resume-Client-NoResume fails for me. Perhaps something
>>> was fixed between master and 1.1.0-pre2.
>>>
>>> David
>>>
>>>
>>>> Thank you
>>>> Durga.
>>>>
>>>> On Tue, Mar 8, 2016 at 3:19 AM, David Benjamin <davidben at google.com>
>>>> wrote:
>>>> > Hi folks,
>>>> >
>>>> > So, we've by now built up a decent-sized SSL test suite in BoringSSL.
>>>> I was
>>>> > bored and ran it against OpenSSL master. It revealed a number of
>>>> bugs. One
>>>> > is https://github.com/openssl/openssl/pull/603. I'll be filing
>>>> tickets
>>>> > shortly for the remaining ones I've triaged, but I thought I'd send
>>>> this
>>>> > separately rather than duplicate it everywhere.
>>>> >
>>>> > Emilia also suggested there may be room to collaborate on testing. If
>>>> > nothing else, just borrowing ideas or porting tests to/from your
>>>> TLSProxy
>>>> > setup. (Like, say, the ones that caught the bugs I'll be reporting.
>>>> :-) )
>>>> > So, here's an introduction on how it all works:
>>>> >
>>>> > To run the tests on OpenSSL, clone BoringSSL:
>>>> > https://boringssl.googlesource.com/boringssl/
>>>> > Then patch in this change. (Click the "Download" in the upper-right
>>>> for
>>>> > options.)
>>>> > https://boringssl-review.googlesource.com/#/c/7332/
>>>> > Then follow the instructions in the commit message.
>>>> >
>>>> > The tests themselves and the runner logic live in
>>>> ssl/test/runner/runner.go:
>>>> >
>>>> https://boringssl.googlesource.com/boringssl/+/22ce9b2d08a52e399bf2ab86851952d727be034d/ssl/test/runner/runner.go#922
>>>> >
>>>> > They work by running an unmodified TLS stack in a shim binary against
>>>> a copy
>>>> > of Go's. We patch our copy with options for weird behavior to test
>>>> against:
>>>> >
>>>> https://boringssl.googlesource.com/boringssl/+/22ce9b2d08a52e399bf2ab86851952d727be034d/ssl/test/runner/common.go#414
>>>> >
>>>> > Go and shim communicate entirely with sockets and (tons of)
>>>> command-line
>>>> > flags, though it is slightly overfit to BoringSSL's behavior and
>>>> checks
>>>> > error strings a lot. The shim also has options like -async mode which
>>>> we use
>>>> > on a subset of tests to stress state machine resumption. (This has
>>>> saved me
>>>> > from state machine bugs so many times.)
>>>> >
>>>> https://boringssl.googlesource.com/boringssl/+/22ce9b2d08a52e399bf2ab86851952d727be034d/ssl/test/runner/runner.go#2770
>>>> >
>>>> https://boringssl.googlesource.com/boringssl/+/22ce9b2d08a52e399bf2ab86851952d727be034d/ssl/test/bssl_shim.cc#826
>>>> >
>>>> > I hope this is useful! Bugs and patches will follow this mail, as I
>>>> write
>>>> > them up.
>>>> >
>>>> > David
>>>> >
>>>> > --
>>>> > openssl-dev mailing list
>>>> > To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev
>>>> >
>>>> --
>>>> openssl-dev mailing list
>>>> To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev
>>>>
>>>
>>> --
>>> openssl-dev mailing list
>>> To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev
>>>
>>>
>> --
>> openssl-dev mailing list
>> To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev
>>
> --
> openssl-dev mailing list
> To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mta.openssl.org/pipermail/openssl-dev/attachments/20160318/7d21f143/attachment.html>