Real MTU problems with BIO pair

[Detlef Vollmann]

On 2020-08-20 21:44, Detlef Vollmann wrote:
> if I create a BIO pair with
>    BIO_new_bio_pair(&int_bio, 0, &ext_bio_, 0);
> 
> then I tried to use SSL_set_mtu(), DTLS_set_link_mtu()
> and SSL_CTX_set_max_send_fragment(ctx, 1000).
> None of them gave me an error, but also none of them worked:
> the ServerHello was still sent as a single packet (>1500 bytes).
It turned out that this was not true: it actually were two
packets but written to the BIO together before SSL_accept()
returned, so my side of the bio pair got on a BIO_read()
one single big packet and sent it to the socket and the wire
as one UDP packet.

> If I create the BIO pair using
>    BIO_new_bio_pair(&int_bio, 1000, &ext_bio_, 1000);
> then the ServerHello is fragmented, but not into DTLS
> handshake fragments, but just into separate UDP packets,
> that neither s_client nor my own client can work with.
> 
> Is there any way to set the maximum fragment size for
> DTLS handshake with a BIO pair?
One solution is to set the MTU and the int_bio size to
exactly the same value.
Another option would be to use BIO_set_callback_ex() and send
the data to the socket after each BIO_write() into int_bio,
but the problem here is that BIO_set_data() cannot be used
as the ptr is already used for the peer address.

   Detlef