BIO_read fails when performing DTLS handshake

[Matt Caswell]

On 31/08/2019 19:40, Matthew wrote:
> Hi Michael,
> 
> 
> Of course I should have provided a version: 1.1.1c, on linux-x64 (dotnet core 2.2)
> 
> Just a quick question while I prepare a clearer, more contextual and concise
> email about my problem;
> 
> Is it normal for an SSL's readBio to be empty /right/ after DTLSv1_listen?
> 
> I noticed that after calling DTLSv1_listen, and it returning 1, the SSL's
> readBio's bytes pending = 0. Would this not force the client to send a second
> hello with cookie? Or does DTLSv1_listen perform the clientHello processing and
> then let SSL_accept send the response?

DTLSv1_listen() (in 1.1.1c) does the following (simplified):

- Listens for incoming ClientHellos
- Reads the ClientHello from the underlying BIO
- Checks to see if the ClientHello contains a cookie or not
- If the cookie is NOT present then it sends a HelloVerifyRequest back and goes
back to the first step above
- Otherwise it verifies the coolie. If verification fails then it sends a
HelloVerifyRequest and goes back to the first step above.
- Otherwise if verification succeeds then it buffers the received ClientHello
record internally in the SSL object and returns 1

So it is normal after DTLSv1_listen() for the BIO to be empty because it has
already read the ClientHello and has buffered it internally.

Note that running DTLS over a mem bio may not always work as expected. Mem bio's
are implemented conceptually as a stream. Therefore they do not respect packet
boundaries and so you may get unusual behaviour. Additionally it doesn't support
any of the datagram related ctrls that a dgram BIO supports. Therefore it can't
do things like identify the address of the peer and query the MTU of the link.

In other words I would recommend using dgram bio rather than a mem bio.

Matt

> 
> On 2019-08-31 10:51 a.m., Michael Wojcik wrote:
>>> From: openssl-users [mailto:openssl-users-bounces at openssl.org] On Behalf Of Matthew
>>> Sent: Friday, August 30, 2019 23:06
>> Welcome to the list. When posting, please remember to tell us what version of OpenSSL you're using, and what platform you're on.
>>
>> Since you're talking about C#, I'll assume the platform is Windows. And I'll assume you're working with OpenSSL 1.1.1c, because that would be the sensible thing to do. But it would be better if I didn't have to make either assumption.
>>
>>> I stepped through both the working unit test and the non-working one in order to find
>>> differences in the result. What I have found is that, in ssl3_read_n, the call to
>>> BIO_read (line 300 in rec_layer_s3.c) returns -1.
>>> ret = BIO_read(s->rbio, pkt + len + left, max - left);
>>> At this line, pkt is a char[8], len and left = 0 and max = 16717
>> I don't think pkt is a char[8]. It's defined at the top of ssl3_read_n as unsigne char *pkt.
>>
>> And it had better not be a char[8], since 1) plain char and unsigned char are not the same type, and 2) if max - left is 16717, then you have potential for a massive buffer overflow.
>>
>>> I'm curious as to why the "data" argument is not a pointer to a buffer, but rather
>>> the result of an addition. Maybe my C isnt strong enough...
>> It's a pointer into a buffer (specifically, in this case, a pointer to the start of a buffer). In C, adding an integer type to a pointer results in a pointer value. That is, in fact, basic C.
>>
>> (At least you're only reading C, not writing it. I have in recent days seen C code posted by people who really need to put the language down and back away slowly. C should not be used by people who don't know the language very well.)
>>
>>> Going even further down the stack, I finally end up at the bottom:
>>> static int mem_read(BIO *b, char *out, int outl)
>>> ...
>>> } else if (bm->length == 0) {
>>>     ret = b->num;
>>>     if (ret != 0)
>>>         BIO_set_retry_read(b);
>>> }
>>> return ret;
>>> At this point, ret = -1...
>> So b->num == -1 when you arrived here with bm->length == 0.
>>
>> The num field is initialized to -1 when a memory BIO is initialized (mem_init in bss_mem.c). And the length is 0, which means there's no data in the BIO.
>>
>> I don't know (without reading through your code, which I don't have time to do right now) why you're using a memory BIO, or how you've initialized it. It looks like you've simply never put any data into it.
>>
>> --
>> Michael Wojcik
>> Distinguished Engineer, Micro Focus
>>
>>
>>