[EXT] Re: KTLS with openssl 3.0 fail with error ENOTCONN(Transport endpoint is not connected)

[Gaurav Jain]

Hi John/Matt

> -----Original Message-----
> From: John Baldwin <jhb at FreeBSD.org>
> Sent: Thursday, January 6, 2022 11:21 PM
> To: Gaurav Jain <gaurav.jain at nxp.com>; borisp at mellanox.com; openssl-
> users at openssl.org
> Cc: Varun Sethi <V.Sethi at nxp.com>; Pankaj Gupta <pankaj.gupta at nxp.com>
> Subject: Re: [EXT] Re: KTLS with openssl 3.0 fail with error
> ENOTCONN(Transport endpoint is not connected)
> 
> Caution: EXT Email
> 
> On 1/6/22 5:58 AM, Gaurav Jain wrote:
> > Hi
> >
> >> -----Original Message-----
> >> From: John Baldwin <jhb at FreeBSD.org>
> >> Sent: Thursday, January 6, 2022 12:26 AM
> >> To: Gaurav Jain <gaurav.jain at nxp.com>; borisp at mellanox.com; openssl-
> >> users at openssl.org
> >> Cc: Varun Sethi <V.Sethi at nxp.com>; Pankaj Gupta
> >> <pankaj.gupta at nxp.com>
> >> Subject: [EXT] Re: KTLS with openssl 3.0 fail with error
> >> ENOTCONN(Transport endpoint is not connected)
> >>
> >> Caution: EXT Email
> >>
> >> On 1/4/22 11:49 PM, Gaurav Jain wrote:
> >>> Hello Boris/John
> >>>
> >>> I am from NXP and currently working on enabling KTLS on NXP
> >>> platforms via
> >> openssl.
> >>> I see that you enabled KTLS support in openssl
> >> 3.0(https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2
> >> Fwww
> >> .openssl.org%2Fnews%2Fchangelog.html%23openssl-
> >>
> 30&data=04%7C01%7Cgaurav.jain%40nxp.com%7Ce87da43a5488475b2aa
> >>
> d08d9d07d05b0%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C1%7C63777
> >>
> 0057654781203%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQ
> >>
> IjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=vZa0aCu
> >> D%2FzrXB0vv23DZiOWSVichep42YLqA4a1JeXY%3D&reserved=0).
> >>>
> >>> when I configure openssl 3.0 or 3.1.0 with enable-ktls and and try
> >>> to run the
> >> s_server, s_client application.
> >>> I observe that connection is successfully established - but it didn't use KTLS.
> >>>
> >>> Then I added additional log in kernel(file net/tls/tls_main.c) and
> >>> see that kernel is returning error -ENOTCONN when (sk->sk_state !=
> >>> TCP_ESTABLISHED) in function static int tls_init(struct sock *sk)
> >>
> >> To be clear, I have worked on KTLS support for FreeBSD, not for Linux.
> >>
> >> However, I think the error you are seeing is a red herring.  I think
> >> you are seeing the setsockopt() call from ktls_enable() fail because
> >> it is invoked on the listen socket since ktls_enable() is called when sockets are
> created by libssl.
> >>
> >> For KTLS to work on the server side on Linux what you need to find
> >> out is when
> >> ktls_enable() is invoked on the socket returned by accept() and why that is
> failing.
> >>
> >
> > Thanks John for your input.
> > Ktls_enable() after accept() is successful on server side.
> > I added debug logs, ktls_start() is failing with error Invalid argument.
> >
> > Logs:
> > openssl s_server -ktls -key rsa.key -cert server.pem -accept 443
> >
> > Using default temp DH parameters
> >
> > ACCEPT
> >
> > ktls_enable setsockopt success, ret = 0
> >
> > ktls_enable() = 1
> >
> >
> > fd = 4, is_tx = 0, tls_crypto_info_len = 1872610871009456445
> >
> > ktls_start setsockopt failed, 22, Invalid argument
> >
> > fd = 4, is_tx = 2, tls_crypto_info_len = 8329596950154514032
> >
> > ktls_start setsockopt failed, 22, Invalid argument
> 
> You'd have to add traces to see why this is failing I think.  I know on FreeBSD we
> will fail the equivalent setsockopt if the ciphersuite isn't supported (e.g.
> some cipher suites we only support if the socket is routed over a NIC with
> offload support, and there can also be system tunables to disable certain cipher
> suites).
> I don't quite know what that is like on the Linux side though.
> 
> --
> John Baldwin

I debugged further the invalid argument error reported from setsockopt called in ktls_start().
Problem is with crypto_info->tls_crypto_info_len.
when I applied the below change I could run s_server, s_client on linux with KTLS support. 
below patch enables the aes_gcm_128 ciphers 
1. AES128-GCM-SHA256 
2. ECDHE-RSA-AES128-GCM-SHA256

diff --git a/include/internal/ktls.h b/include/internal/ktls.h index 95492fd065..30ed5072de 100644 - a/include/internal/ktls.h
+++ b/include/internal/ktls.h
@@ -285,7 +285,7 @@ static ossl_inline int ktls_start(int fd, ktls_crypto_info_t *crypto_info, int is_tx)  {
     return setsockopt(fd, SOL_TLS, is_tx ? TLS_TX : TLS_RX,
-                     crypto_info, crypto_info->tls_crypto_info_len) ? 0 : 1;
+                      crypto_info, sizeof(crypto_info->gcm128)) ? 0 : 1;
 }

When trying to enable other 2 ciphers, I am getting build errors in openssl.
3. AES256-GCM-SHA384
4. ECDHE-RSA-AES256-GCM-SHA384
when trying to build openssl for gcm256, getting compilation errors:

In file included from crypto/bio/bio_sock2.c:15:
include/internal/ktls.h:251:46: error: field 'gcm256' has incomplete type
         struct tls12_crypto_info_aes_gcm_256 gcm256;
                                              ^~~~~~ 

By default Openssl has support for  aes_gcm_128 ciphers.
When include kernel tls.h file in openssl, aes_gcm_256 ciphers can be built and run without errors.

I feel that Openssl KTLS support for Linux is broken.

Regards
Gaurav Jain