[openssl-dev] [RFC 0/4] Kernel TLS socket API

[Boris Pismenny]

Hello all,

I would like to introduce you to the new kernel API for TLS transmit-side
data-path, and open a discussion regarding its support in OpenSSL.

This is currently a V2 patch series in Linux net-next, and it is stabilizing.
Dave has been working on this for a while [1][2], and Aviad, Ilya and myself
have also been working on TLS transmit-side crypto offload to a NIC [3][4].
Similar work is also being done in FreeBSD by Netflix [5][6].

Placing the data-path of TLS in the kernel has several advantages. First, an
efficient TLS sendfile system call implementation is possible with such a
socket. Second, data from userspace could be encrypted when passing the
user-kernel boundary, reducing the overhead of existing implementations that
require at least encrypting the data and then copying from user to kernel.
Third, TLS data-path crypto offload to hardware requires a kernel
infrastructure for TLS.

To sum up, a new kernel socket option is introduced to add support for the
data-path of TLS and it will soon be common in operating systems.

For anyone who is interested, here is the kernel code:
https://github.com/Mellanox/tls-offload - branch tx_rfc_v10
And our openssl development branch:
https://github.com/Mellanox/tls-openssl - branch tls_tx_v2

The user API is as follows: (also see [7])
==========================================

Creating a TLS connection
-------------------------
First create a new TCP socket and set the TLS ULP.

  sock = socket(AF_INET, SOCK_STREAM, 0);
  setsockopt(sock, SOL_TCP, TCP_ULP, "tls", sizeof("tls"));

Setting the TLS ULP allows us to set/get TLS socket options. 
Currently only the symmetric encryption is handled in the kernel. Patch 2 sets
the TLS ULP in BIO_new_socket.

All the parameters required to move the data-path to the kernel are available
after processing the change cipher state message during the handshake.
Patch 4 adds code similar to the one below in tls1_change_cipher_state.

There is a separate socket option for moving the transmit and the receive into
the kernel. Current code, supports only the transmit side, TLS1.2 and
AES-GCM128. In the future, this will be extended.

Below is the code that sets the socket option which moves the transmit
data-path into the kernel. Patch 1 adds a temporary netinet/tcp.h file which
includes these structures. Do you know if netinet/tcp.h is the right place for
these?

 /* From linux/tls.h */
 struct tls_crypto_info {
         unsigned short version;
         unsigned short cipher_type;
 };

  struct tls12_crypto_info_aes_gcm_128 {
          struct tls_crypto_info info;
          unsigned char iv[TLS_CIPHER_AES_GCM_128_IV_SIZE];
          unsigned char key[TLS_CIPHER_AES_GCM_128_KEY_SIZE];
          unsigned char salt[TLS_CIPHER_AES_GCM_128_SALT_SIZE];
          unsigned char rec_seq[TLS_CIPHER_AES_GCM_128_REC_SEQ_SIZE];
  };


  struct tls12_crypto_info_aes_gcm_128 crypto_info;

  crypto_info.info.version = TLS_1_2_VERSION;
  crypto_info.info.cipher_type = TLS_CIPHER_AES_GCM_128;
  memcpy(crypto_info.iv, iv_write, TLS_CIPHER_AES_GCM_128_IV_SIZE);
  memcpy(crypto_info.rec_seq, seq_number_write,
					TLS_CIPHER_AES_GCM_128_REC_SEQ_SIZE);
  memcpy(crypto_info.key, cipher_key_write, TLS_CIPHER_AES_GCM_128_KEY_SIZE);
  memcpy(crypto_info.salt, implicit_iv_write, TLS_CIPHER_AES_GCM_128_SALT_SIZE);

  setsockopt(sock, SOL_TLS, TLS_TX, &crypto_info, sizeof(crypto_info));

Sending TLS application data
----------------------------

After setting the TLS_TX socket option all application data sent over this
socket is encrypted using TLS and the parameters provided in the socket option.

For example, an application can send an encrypted hello world record as follows:

  const char *msg = "hello world\n";
  send(sock, msg, strlen(msg));

send() data is directly encrypted from the userspace buffer provided
to the encrypted kernel send buffer if possible.

The sendfile system call will send the file's data over TLS records of maximum
length (2^14).

  file = open(filename, O_RDONLY);
  fstat(file, &stat);
  sendfile(sock, file, &offset, stat.st_size);

TLS records are created and sent after each send() call, unless
MSG_MORE is passed.  MSG_MORE will delay creation of a record until
MSG_MORE is not passed, or the maximum record size is reached
or an alert record needs to be sent.

The kernel will need to allocate a buffer for the encrypted data.
This buffer is allocated at the time send() is called, such that
either the entire send() call will return -ENOMEM (or block waiting
for memory), or the encryption will always succeed.  If send() returns -ENOMEM
and some data was left on the socket buffer from a previous call using
MSG_MORE, the MSG_MORE data is left on the socket buffer.

Send TLS control messages
-------------------------

Other than application data, TLS has control messages such as alert messages
(record type 21) and handshake messages (record type 22), etc. These messages
can be sent over the socket by providing the TLS record type via a CMSG.
Patch 2 adds the following function (in bss_sock.c) that sends @data of @length
bytes using a record of type @record_type.

/* send TLS control message using record_type */
  static int klts_send_ctrl_message(int sock, unsigned char record_type,
                                  void *data, size_t length)
  {
        struct msghdr msg = {0};
        int cmsg_len = sizeof(record_type);
        struct cmsghdr *cmsg;
        char buf[CMSG_SPACE(cmsg_len)];
        struct iovec msg_iov;   /* Vector of data to send/receive into.  */

        msg.msg_control = buf;
        msg.msg_controllen = sizeof(buf);
        cmsg = CMSG_FIRSTHDR(&msg);
        cmsg->cmsg_level = SOL_TLS;
        cmsg->cmsg_type = TLS_SET_RECORD_TYPE;
        cmsg->cmsg_len = CMSG_LEN(cmsg_len);
        *CMSG_DATA(cmsg) = record_type;
        msg.msg_controllen = cmsg->cmsg_len;

        msg_iov.iov_base = data;
        msg_iov.iov_len = length;
        msg.msg_iov = &msg_iov;
        msg.msg_iovlen = 1;

        return sendmsg(sock, &msg, 0);
  }

Control message data should be provided unencrypted, and will be
encrypted by the kernel.  At a high level, the kernel TLS ULP is a
replacement for the record layer of a userspace TLS library.

References:
[1] - https://netdevconf.org/1.2/session.html?dave-watson
[2] - https://www.spinics.net/lists/linux-crypto/msg25953.html
[3] - https://netdevconf.org/1.2/session.html?boris-pismenny
[4] - https://www.spinics.net/lists/netdev/msg427122.html
[5] - https://people.freebsd.org/~rrs/asiabsd_2015_tls.pdf
[6] - https://people.freebsd.org/~rrs/asiabsd_tls_improved.pdf
[7] - https://www.spinics.net/lists/linux-crypto/msg25957.html

Boris Pismenny (4):
  e_os: hack! TLS offload
  bio: Linux TLS Offload
  evp/e_aes: Expose GCM IV
  ssl: Linux TLS Tx Offload

 crypto/bio/bss_sock.c     |  93 +++++++++++++-
 crypto/evp/e_aes.c        |   8 ++
 e_os.h                    |   2 +-
 include/netinet/tcp.h     | 305 ++++++++++++++++++++++++++++++++++++++++++++++
 include/openssl/bio.h     |  32 +++++
 include/openssl/evp.h     |   1 +
 ssl/record/rec_layer_s3.c |  95 +++++++++++----
 ssl/t1_enc.c              |  69 +++++++++++
 8 files changed, 578 insertions(+), 27 deletions(-)
 create mode 100644 include/netinet/tcp.h

-- 
1.8.3.1