[openssl-dev] Running against BoringSSL's SSL test suite

[David Benjamin]

On Thu, Mar 10, 2016 at 1:30 AM Kanaka Kotamarthy <kotamarthyd at gmail.com>
wrote:

> And also Openssl fails with Resume-Client-NoResume cases. Do you have
> any report on which test cases do fail and reasons for the failure?
>
>
> RT tickets 4387 through 4395 were the failures I've triaged. I'm sure
> there's more things in there to look through.
>
> I don't believe Resume-Client-NoResume fails for me. Perhaps something was
> fixed between master and 1.1.0-pre2.
>
>
> Openssl doesn't gives any error. For Resume-Client-NoResume-SSL3-TLS11
> test case, we expect the new session's handshake to be done with TLS11. But
> with Openssl handshake is done using SSL3. As in ssl3_clear, we set back
> s->version to s->method->version.
>

Oh, sorry, I keep forgetting our runner doesn't make it clear when a -test
option fails to match anything. (I should fix that...) I looked
for Resume-Client-NoResume without noticing it had suffixes. :-)

I would expect most things addResumptionVersionTests to fail. See
https://github.com/openssl/openssl/pull/603

David


> Thank you
> Durga.
>
> On Wed, Mar 9, 2016 at 10:38 PM, David Benjamin <davidben at google.com>
> wrote:
>
>> On Wed, Mar 9, 2016 at 5:07 AM Kanaka Kotamarthy <kotamarthyd at gmail.com>
>> wrote:
>>
>>> Hi
>>>
>>> I am even testing OpenSSL with BoringSSL's test cases using
>>> Openssl-1.1.0-pre2. Trying to find out reasons of OpenSSL's failures
>>> for particular cases.
>>>
>>> DTLS 1.0 session resumption has some thing wrong. If s_server started
>>> with -dtls and s_client -dtls1 -reconnect , session resumption is not
>>> being done. The reason for this may be, version negotiation for DTLS
>>> is done after loading previous session and check for s->version and
>>> s->session->version fails in tls_process_client_hello.
>>>
>>
>> See RT #4392.
>> https://rt.openssl.org/Ticket/Display.html?id=4392
>>
>>
>>> And also Openssl fails with Resume-Client-NoResume cases. Do you have
>>> any report on which test cases do fail and reasons for the failure?
>>>
>>
>> RT tickets 4387 through 4395 were the failures I've triaged. I'm sure
>> there's more things in there to look through.
>>
>> I don't believe Resume-Client-NoResume fails for me. Perhaps something
>> was fixed between master and 1.1.0-pre2.
>>
>> David
>>
>>
>>> Thank you
>>> Durga.
>>>
>>> On Tue, Mar 8, 2016 at 3:19 AM, David Benjamin <davidben at google.com>
>>> wrote:
>>> > Hi folks,
>>> >
>>> > So, we've by now built up a decent-sized SSL test suite in BoringSSL.
>>> I was
>>> > bored and ran it against OpenSSL master. It revealed a number of bugs.
>>> One
>>> > is https://github.com/openssl/openssl/pull/603. I'll be filing tickets
>>> > shortly for the remaining ones I've triaged, but I thought I'd send
>>> this
>>> > separately rather than duplicate it everywhere.
>>> >
>>> > Emilia also suggested there may be room to collaborate on testing. If
>>> > nothing else, just borrowing ideas or porting tests to/from your
>>> TLSProxy
>>> > setup. (Like, say, the ones that caught the bugs I'll be reporting.
>>> :-) )
>>> > So, here's an introduction on how it all works:
>>> >
>>> > To run the tests on OpenSSL, clone BoringSSL:
>>> > https://boringssl.googlesource.com/boringssl/
>>> > Then patch in this change. (Click the "Download" in the upper-right for
>>> > options.)
>>> > https://boringssl-review.googlesource.com/#/c/7332/
>>> > Then follow the instructions in the commit message.
>>> >
>>> > The tests themselves and the runner logic live in
>>> ssl/test/runner/runner.go:
>>> >
>>> https://boringssl.googlesource.com/boringssl/+/22ce9b2d08a52e399bf2ab86851952d727be034d/ssl/test/runner/runner.go#922
>>> >
>>> > They work by running an unmodified TLS stack in a shim binary against
>>> a copy
>>> > of Go's. We patch our copy with options for weird behavior to test
>>> against:
>>> >
>>> https://boringssl.googlesource.com/boringssl/+/22ce9b2d08a52e399bf2ab86851952d727be034d/ssl/test/runner/common.go#414
>>> >
>>> > Go and shim communicate entirely with sockets and (tons of)
>>> command-line
>>> > flags, though it is slightly overfit to BoringSSL's behavior and checks
>>> > error strings a lot. The shim also has options like -async mode which
>>> we use
>>> > on a subset of tests to stress state machine resumption. (This has
>>> saved me
>>> > from state machine bugs so many times.)
>>> >
>>> https://boringssl.googlesource.com/boringssl/+/22ce9b2d08a52e399bf2ab86851952d727be034d/ssl/test/runner/runner.go#2770
>>> >
>>> https://boringssl.googlesource.com/boringssl/+/22ce9b2d08a52e399bf2ab86851952d727be034d/ssl/test/bssl_shim.cc#826
>>> >
>>> > I hope this is useful! Bugs and patches will follow this mail, as I
>>> write
>>> > them up.
>>> >
>>> > David
>>> >
>>> > --
>>> > openssl-dev mailing list
>>> > To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev
>>> >
>>> --
>>> openssl-dev mailing list
>>> To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev
>>>
>>
>> --
>> openssl-dev mailing list
>> To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev
>>
>>
> --
> openssl-dev mailing list
> To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mta.openssl.org/pipermail/openssl-dev/attachments/20160310/de82dee1/attachment.html>