[openssl-dev] Calculating DTLS payload MTU

[David Woodhouse]

On Wed, 2016-10-05 at 10:04 +0100, Matt Caswell wrote:
> 
> > For example for AES-128-GCM-SHA256 we'd start with 1500 and subtract:
> >  - 20 bytes for a Legacy IP header.
> >  - 8 bytes for UDP header.
> >  - 13 bytes for DTLS header
> >  - 16 bytes for the hash
> >  - 8 bytes for nonce
> > 
> > ... and be left with 1435 bytes.
> > 
> > In GnuTLS this is fairly trivial; I call gnutls_dtls_set_mtu() followed
> > by gnutls_dtls_get_data_mtu().
> > 
> > How do I do it in OpenSSL? Do I need to build a big table of the
> > overhead of all ciphers and calculate it for myself?
> 
> I don't think there is a simple way to do this.
> 
> You can ask the underlying BIO to give you the transport protocol
> overhead using BIO_dgram_get_mtu_overhead(). DTLS1_RT_HEADER_LENGTH
> gives you the DTLS header value. You can find out features of the
> ciphersuite using SSL_get_cipher().

Right, it's the "features of the ciphersuite" which is the fun part. 

Specifically, the size of the nonce/IV, the size of the hash, and the
block size for block ciphers where we need to round up the sizes.

SSL_get_cipher() gives me a string, and I could implement a big lookup
table — but we recently moved to using PSK and "proper" DTLS
negotiation precisely to *avoid* having to have ciphersuite-specific
knowledge in the client.

How's this for a start... how is it that the more I work on this stuff,
the more I realise how *utterly* clueless about it I am? :)

Specific questions I *know* I need to focus on include which *other*
modes other than CBC/CCM/GCM I might need to handle, and fairly much
everything about the CCM case.

Can we add something to OpenSSL 1.2 which means I don't need to keep
doing this in the application, please?

/* sets the DTLS MTU and returns the actual tunnel MTU */
unsigned dtls_set_mtu(struct openconnect_info *vpninfo, unsigned mtu)
{
	int tun_mtu;
	int ivlen, maclen, blocksize = 1, pad = 0;

#if OPENSSL_VERSION_NUMBER < 0x10100000L || defined(LIBRESSL_VERSION_NUMBER)
	/* OpenSSL <= 1.0.2 only supports CBC ciphers with PSK */
	ivlen = EVP_CIPHER_iv_length(EVP_CIPHER_CTX_cipher(vpninfo->dtls_ssl->enc_write_ctx));
	maclen = EVP_MD_CTX_size(vpninfo->dtls_ssl->write_hash);
	blocksize = ivlen;
	pad = 1;
#else
	/* Now it gets more fun... */
	const SSL_CIPHER *s_ciph = SSL_get_current_cipher(vpninfo->dtls_ssl);
	const EVP_CIPHER *e_ciph;
	const EVP_MD *e_md;

	e_ciph = EVP_get_cipherbynid(SSL_CIPHER_get_cipher_nid(s_ciph));

	switch (EVP_CIPHER_mode(e_ciph)) {
	case EVP_CIPH_GCM_MODE:
		ivlen = EVP_GCM_TLS_EXPLICIT_IV_LEN;
		maclen = EVP_GCM_TLS_TAG_LEN;
		blocksize = 1;
		pad = 0;
		break;

	case EVP_CIPH_CCM_MODE:
		ivlen = EVP_CCM_TLS_EXPLICIT_IV_LEN;
		/* What is the tag size for CCM? */
		maclen = blocksize = EVP_CIPHER_block_size(e_ciph);
		/* Padding same as CBC? */
		pad = 1;
		break;

	case EVP_CIPH_CBC_MODE:
		e_md = EVP_get_digestbynid(SSL_CIPHER_get_digest_nid(s_ciph));
		blocksize = EVP_CIPHER_block_size(e_ciph);
		ivlen = EVP_CIPHER_iv_length(e_ciph);
		pad = 1;
		maclen = EVP_MD_size(e_md);
		break;

	default:
		// XXX
		;
	}
#endif


	/* Take off the explicit IV and the MAC (XXX: overflow!) */
	printf("iv %d mac %d blk %d\n", ivlen, maclen, blocksize);
	tun_mtu = mtu - DTLS1_RT_HEADER_LENGTH - ivlen - maclen;
	/* For block cipher modes round down to blocksize */
	printf("tun %d & 0x%x == %d\n", tun_mtu, ~(blocksize-1), tun_mtu & (~(blocksize-1)));
	tun_mtu &= ~(blocksize-1);
	/* ... and CBC modes require at least one byte to indicate padding length */
	tun_mtu -= pad;

	DTLS_set_link_mtu(vpninfo->dtls_ssl, mtu);

	/* We already set the link MTU, but hopefully by the time we
	 * finish it, this function will be better at working out the
	 * actual tunnel MTU than OpenSSL is. So do that too... */
	SSL_set_mtu(vpninfo->dtls_ssl, tun_mtu);

	return tun_mtu;
}

-- 
dwmw2
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 5760 bytes
Desc: not available
URL: <http://mta.openssl.org/pipermail/openssl-dev/attachments/20161005/9b0bf8f2/attachment.bin>