BIO_read fails when performing DTLS handshake

[Matthew]

Hi Michael,


Of course I should have provided a version: 1.1.1c, on linux-x64 (dotnet 
core 2.2)

Just a quick question while I prepare a clearer, more contextual and 
concise email about my problem;

Is it normal for an SSL's readBio to be empty /right/ after DTLSv1_listen?

I noticed that after calling DTLSv1_listen, and it returning 1, the 
SSL's readBio's bytes pending = 0. Would this not force the client to 
send a second hello with cookie? Or does DTLSv1_listen perform the 
clientHello processing and then let SSL_accept send the response?

On 2019-08-31 10:51 a.m., Michael Wojcik wrote:
>> From: openssl-users [mailto:openssl-users-bounces at openssl.org] On Behalf Of Matthew
>> Sent: Friday, August 30, 2019 23:06
> Welcome to the list. When posting, please remember to tell us what version of OpenSSL you're using, and what platform you're on.
>
> Since you're talking about C#, I'll assume the platform is Windows. And I'll assume you're working with OpenSSL 1.1.1c, because that would be the sensible thing to do. But it would be better if I didn't have to make either assumption.
>
>> I stepped through both the working unit test and the non-working one in order to find
>> differences in the result. What I have found is that, in ssl3_read_n, the call to
>> BIO_read (line 300 in rec_layer_s3.c) returns -1.
>> ret = BIO_read(s->rbio, pkt + len + left, max - left);
>> At this line, pkt is a char[8], len and left = 0 and max = 16717
> I don't think pkt is a char[8]. It's defined at the top of ssl3_read_n as unsigne char *pkt.
>
> And it had better not be a char[8], since 1) plain char and unsigned char are not the same type, and 2) if max - left is 16717, then you have potential for a massive buffer overflow.
>
>> I'm curious as to why the "data" argument is not a pointer to a buffer, but rather
>> the result of an addition. Maybe my C isnt strong enough...
> It's a pointer into a buffer (specifically, in this case, a pointer to the start of a buffer). In C, adding an integer type to a pointer results in a pointer value. That is, in fact, basic C.
>
> (At least you're only reading C, not writing it. I have in recent days seen C code posted by people who really need to put the language down and back away slowly. C should not be used by people who don't know the language very well.)
>
>> Going even further down the stack, I finally end up at the bottom:
>> static int mem_read(BIO *b, char *out, int outl)
>> ...
>> } else if (bm->length == 0) {
>>      ret = b->num;
>>      if (ret != 0)
>>          BIO_set_retry_read(b);
>> }
>> return ret;
>> At this point, ret = -1...
> So b->num == -1 when you arrived here with bm->length == 0.
>
> The num field is initialized to -1 when a memory BIO is initialized (mem_init in bss_mem.c). And the length is 0, which means there's no data in the BIO.
>
> I don't know (without reading through your code, which I don't have time to do right now) why you're using a memory BIO, or how you've initialized it. It looks like you've simply never put any data into it.
>
> --
> Michael Wojcik
> Distinguished Engineer, Micro Focus
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mta.openssl.org/pipermail/openssl-users/attachments/20190831/dde40328/attachment.html>