BIO_read fails when performing DTLS handshake

[Michael Wojcik]

> From: openssl-users [mailto:openssl-users-bounces at openssl.org] On Behalf Of Matthew
> Sent: Friday, August 30, 2019 23:06

Welcome to the list. When posting, please remember to tell us what version of OpenSSL you're using, and what platform you're on.

Since you're talking about C#, I'll assume the platform is Windows. And I'll assume you're working with OpenSSL 1.1.1c, because that would be the sensible thing to do. But it would be better if I didn't have to make either assumption.

> I stepped through both the working unit test and the non-working one in order to find
> differences in the result. What I have found is that, in ssl3_read_n, the call to
> BIO_read (line 300 in rec_layer_s3.c) returns -1.
> ret = BIO_read(s->rbio, pkt + len + left, max - left);

> At this line, pkt is a char[8], len and left = 0 and max = 16717

I don't think pkt is a char[8]. It's defined at the top of ssl3_read_n as unsigne char *pkt.

And it had better not be a char[8], since 1) plain char and unsigned char are not the same type, and 2) if max - left is 16717, then you have potential for a massive buffer overflow.

> I'm curious as to why the "data" argument is not a pointer to a buffer, but rather
> the result of an addition. Maybe my C isnt strong enough...

It's a pointer into a buffer (specifically, in this case, a pointer to the start of a buffer). In C, adding an integer type to a pointer results in a pointer value. That is, in fact, basic C.

(At least you're only reading C, not writing it. I have in recent days seen C code posted by people who really need to put the language down and back away slowly. C should not be used by people who don't know the language very well.)

> Going even further down the stack, I finally end up at the bottom:
> static int mem_read(BIO *b, char *out, int outl)
> ...
> } else if (bm->length == 0) {
>     ret = b->num;
>     if (ret != 0)
>         BIO_set_retry_read(b);
> }
> return ret;

> At this point, ret = -1...

So b->num == -1 when you arrived here with bm->length == 0.

The num field is initialized to -1 when a memory BIO is initialized (mem_init in bss_mem.c). And the length is 0, which means there's no data in the BIO.

I don't know (without reading through your code, which I don't have time to do right now) why you're using a memory BIO, or how you've initialized it. It looks like you've simply never put any data into it.

--
Michael Wojcik
Distinguished Engineer, Micro Focus