[openssl-dev] Calculating DTLS payload MTU

[Matt Caswell]

On 05/10/16 00:39, David Woodhouse wrote:
> I have the link MTU (typically 1500 bytes), and a DTLS session is
> established.
> 
> I call DTLS_set_link_mtu() to set the link MTU.
> 
> I need to know the DTLS data MTU — the maximum payload size, which
> depends on the cipher in use.
> 
> For example for AES-128-GCM-SHA256 we'd start with 1500 and subtract:
>  - 20 bytes for a Legacy IP header.
>  - 8 bytes for UDP header.
>  - 13 bytes for DTLS header
>  - 16 bytes for the hash
>  - 8 bytes for nonce
> 
> ... and be left with 1435 bytes.
> 
> In GnuTLS this is fairly trivial; I call gnutls_dtls_set_mtu() followed
> by gnutls_dtls_get_data_mtu().
> 
> How do I do it in OpenSSL? Do I need to build a big table of the
> overhead of all ciphers and calculate it for myself?

I don't think there is a simple way to do this.

You can ask the underlying BIO to give you the transport protocol
overhead using BIO_dgram_get_mtu_overhead(). DTLS1_RT_HEADER_LENGTH
gives you the DTLS header value. You can find out features of the
ciphersuite using SSL_get_cipher().

Matt