[openssl-users] DTLS fragmentation and mem BIO

[Lorenzo Miniero]

Hi all,

first of all, apologies if this has been asked before. I've searched
archives pretty much everywhere, and only came to partial indications as to
how this should be dealt with.

The problem I'm facing deals with using DTLS with mem BIOs, as I have to
take care of transport myself. Specifically, I've implemented a WebRTC
gateway called Janus, which means all the connectivity related stuff is
delegated to another library (libnice in this case). This mostly works
great (kudos to you guys!), but I have problems as soon as packets exceed
the MTU, which can easily happen whenever, for instance, you try to
handshake with certificates larger than 1024 bits. I read around that the
DTLS stack in OpenSSL automatically deals with this, and in fact this seems
to be happening: what isn't working is the BIO mem part of this.

More specifically, OpenSSL does indeed take care of fragmenting the packets
according to what is assumed to be the MTU (1472 by default, or the value
as set in s->d1->mtu). The problem is that the mem BIO ignores that
fragmentation info completely, and so, when you do an BIO_read, makes
available at the application the whole packet anyway. This results in the
whole buffer being passed to nice_agent_send (the method libnice exposes to
send packets), which means it's just as not fragmenting anything: the
packet is too large and the network drops it. You can verify this by using,
e.g., a 4096 bits certificate, and capture the DTLS traffic with Wireshark:
you'll see that the message is recognized as composed of not only multiple
messages, but also fragments.

Is there any way I can force the BIO to return the invididual
fragments/messages when I do a BIO_read, so that I can send properly sized
packets? I've tried looking around but found no insight on how to do that.
The only approach that came to my mind was to manually inspect the buffer
that is returned, and split messages/fragments myself, but I'd rather avoid
delving within the specifics of the protocol if possible.

Thanks in advance for any help you may provide me with!
Lorenzo
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mta.openssl.org/pipermail/openssl-users/attachments/20150605/2c342f73/attachment.html>