[openssl-users] DTLS fragmentation and mem BIO

[Matt Caswell]

On 05/06/15 08:09, Lorenzo Miniero wrote:
> Hi all,
> 
> first of all, apologies if this has been asked before. I've searched
> archives pretty much everywhere, and only came to partial indications as
> to how this should be dealt with.
> 
> The problem I'm facing deals with using DTLS with mem BIOs, as I have to
> take care of transport myself. Specifically, I've implemented a WebRTC
> gateway called Janus, which means all the connectivity related stuff is
> delegated to another library (libnice in this case). This mostly works
> great (kudos to you guys!), but I have problems as soon as packets
> exceed the MTU, which can easily happen whenever, for instance, you try
> to handshake with certificates larger than 1024 bits. I read around that
> the DTLS stack in OpenSSL automatically deals with this, and in fact
> this seems to be happening: what isn't working is the BIO mem part of this.
> 
> More specifically, OpenSSL does indeed take care of fragmenting the
> packets according to what is assumed to be the MTU (1472 by default, or
> the value as set in s->d1->mtu). The problem is that the mem BIO ignores
> that fragmentation info completely, and so, when you do an BIO_read,
> makes available at the application the whole packet anyway. This results
> in the whole buffer being passed to nice_agent_send (the method libnice
> exposes to send packets), which means it's just as not fragmenting
> anything: the packet is too large and the network drops it. You can
> verify this by using, e.g., a 4096 bits certificate, and capture the
> DTLS traffic with Wireshark: you'll see that the message is recognized
> as composed of not only multiple messages, but also fragments.
> 
> Is there any way I can force the BIO to return the invididual
> fragments/messages when I do a BIO_read, so that I can send properly
> sized packets? I've tried looking around but found no insight on how to
> do that. The only approach that came to my mind was to manually inspect
> the buffer that is returned, and split messages/fragments myself, but
> I'd rather avoid delving within the specifics of the protocol if possible.
> 
> Thanks in advance for any help you may provide me with!

Hmmmm. An interesting problem.

The issue is that a mem BIO has no knowledge of datagram semantics
(perhaps we need to add something for OpenSSL 1.1.0).

In a dgram BIO each BIO_write translates to a single datagram being
produced. In a mem BIO you just have a big bucket of memory, and every
time you get a BIO_write you just add the data onto the end of
everything that we've go so far, and so the packet boundaries are not
respected.

How about you create a custom filter BIO? All it would need to do is
proxy all calls down to the underlying mem BIO. Along the way though it
could take note of where the packet boundaries are, so when you call
BIO_read it only gives it to you a datagram at a time.

Matt