[EXT] Re: KTLS with openssl 3.0 fail with error ENOTCONN(Transport endpoint is not connected)

Thu Jan 13 06:20:39 UTC 2022

Hi

> -----Original Message-----
> From: Gaurav Jain
> Sent: Monday, January 10, 2022 1:12 PM
> To: John Baldwin <jhb at FreeBSD.org>; borisp at mellanox.com; openssl-
> users at openssl.org; matt at openssl.org
> Cc: Varun Sethi <V.Sethi at nxp.com>; Pankaj Gupta <pankaj.gupta at nxp.com>
> Subject: RE: [EXT] Re: KTLS with openssl 3.0 fail with error
> ENOTCONN(Transport endpoint is not connected)
> 
> Hi John/Matt
> 
> > -----Original Message-----
> > From: John Baldwin <jhb at FreeBSD.org>
> > Sent: Thursday, January 6, 2022 11:21 PM
> > To: Gaurav Jain <gaurav.jain at nxp.com>; borisp at mellanox.com; openssl-
> > users at openssl.org
> > Cc: Varun Sethi <V.Sethi at nxp.com>; Pankaj Gupta <pankaj.gupta at nxp.com>
> > Subject: Re: [EXT] Re: KTLS with openssl 3.0 fail with error
> > ENOTCONN(Transport endpoint is not connected)
> >
> > Caution: EXT Email
> >
> > On 1/6/22 5:58 AM, Gaurav Jain wrote:
> > > Hi
> > >
> > >> -----Original Message-----
> > >> From: John Baldwin <jhb at FreeBSD.org>
> > >> Sent: Thursday, January 6, 2022 12:26 AM
> > >> To: Gaurav Jain <gaurav.jain at nxp.com>; borisp at mellanox.com;
> > >> openssl- users at openssl.org
> > >> Cc: Varun Sethi <V.Sethi at nxp.com>; Pankaj Gupta
> > >> <pankaj.gupta at nxp.com>
> > >> Subject: [EXT] Re: KTLS with openssl 3.0 fail with error
> > >> ENOTCONN(Transport endpoint is not connected)
> > >>
> > >> Caution: EXT Email
> > >>
> > >> On 1/4/22 11:49 PM, Gaurav Jain wrote:
> > >>> Hello Boris/John
> > >>>
> > >>> I am from NXP and currently working on enabling KTLS on NXP
> > >>> platforms via
> > >> openssl.
> > >>> I see that you enabled KTLS support in openssl
> > >> 3.0(https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F
> > >> %2
> > >> Fwww
> > >> .openssl.org%2Fnews%2Fchangelog.html%23openssl-
> > >>
> >
> 30&data=04%7C01%7Cgaurav.jain%40nxp.com%7Ce87da43a5488475b2aa
> > >>
> >
> d08d9d07d05b0%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C1%7C63777
> > >>
> >
> 0057654781203%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQ
> > >>
> >
> IjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=vZa0aCu
> > >> D%2FzrXB0vv23DZiOWSVichep42YLqA4a1JeXY%3D&reserved=0).
> > >>>
> > >>> when I configure openssl 3.0 or 3.1.0 with enable-ktls and and try
> > >>> to run the
> > >> s_server, s_client application.
> > >>> I observe that connection is successfully established - but it didn't use KTLS.
> > >>>
> > >>> Then I added additional log in kernel(file net/tls/tls_main.c) and
> > >>> see that kernel is returning error -ENOTCONN when (sk->sk_state !=
> > >>> TCP_ESTABLISHED) in function static int tls_init(struct sock *sk)
> > >>
> > >> To be clear, I have worked on KTLS support for FreeBSD, not for Linux.
> > >>
> > >> However, I think the error you are seeing is a red herring.  I
> > >> think you are seeing the setsockopt() call from ktls_enable() fail
> > >> because it is invoked on the listen socket since ktls_enable() is
> > >> called when sockets are
> > created by libssl.
> > >>
> > >> For KTLS to work on the server side on Linux what you need to find
> > >> out is when
> > >> ktls_enable() is invoked on the socket returned by accept() and why
> > >> that is
> > failing.
> > >>
> > >
> > > Thanks John for your input.
> > > Ktls_enable() after accept() is successful on server side.
> > > I added debug logs, ktls_start() is failing with error Invalid argument.
> > >
> > > Logs:
> > > openssl s_server -ktls -key rsa.key -cert server.pem -accept 443
> > >
> > > Using default temp DH parameters
> > >
> > > ACCEPT
> > >
> > > ktls_enable setsockopt success, ret = 0
> > >
> > > ktls_enable() = 1
> > >
> > >
> > > fd = 4, is_tx = 0, tls_crypto_info_len = 1872610871009456445
> > >
> > > ktls_start setsockopt failed, 22, Invalid argument
> > >
> > > fd = 4, is_tx = 2, tls_crypto_info_len = 8329596950154514032
> > >
> > > ktls_start setsockopt failed, 22, Invalid argument
> >
> > You'd have to add traces to see why this is failing I think.  I know
> > on FreeBSD we will fail the equivalent setsockopt if the ciphersuite isn't
> supported (e.g.
> > some cipher suites we only support if the socket is routed over a NIC
> > with offload support, and there can also be system tunables to disable
> > certain cipher suites).
> > I don't quite know what that is like on the Linux side though.
> >
> > --
> > John Baldwin
> 
> I debugged further the invalid argument error reported from setsockopt called in
> ktls_start().
> Problem is with crypto_info->tls_crypto_info_len.
> when I applied the below change I could run s_server, s_client on linux with KTLS
> support.
> below patch enables the aes_gcm_128 ciphers 1. AES128-GCM-SHA256 2.
> ECDHE-RSA-AES128-GCM-SHA256
> 
> diff --git a/include/internal/ktls.h b/include/internal/ktls.h index
> 95492fd065..30ed5072de 100644 - a/include/internal/ktls.h
> +++ b/include/internal/ktls.h
> @@ -285,7 +285,7 @@ static ossl_inline int ktls_start(int fd, ktls_crypto_info_t
> *crypto_info, int is_tx)  {
>      return setsockopt(fd, SOL_TLS, is_tx ? TLS_TX : TLS_RX,
> -                     crypto_info, crypto_info->tls_crypto_info_len) ? 0 : 1;
> +                      crypto_info, sizeof(crypto_info->gcm128)) ? 0 :
> + 1;
>  }
> 
> When trying to enable other 2 ciphers, I am getting build errors in openssl.
> 3. AES256-GCM-SHA384
> 4. ECDHE-RSA-AES256-GCM-SHA384
> when trying to build openssl for gcm256, getting compilation errors:
> 
> In file included from crypto/bio/bio_sock2.c:15:
> include/internal/ktls.h:251:46: error: field 'gcm256' has incomplete type
>          struct tls12_crypto_info_aes_gcm_256 gcm256;
>                                               ^~~~~~
> 
> By default Openssl has support for  aes_gcm_128 ciphers.
> When include kernel tls.h file in openssl, aes_gcm_256 ciphers can be built and
> run without errors.
> 
> I feel that Openssl KTLS support for Linux is broken.
> 

Opened a bug https://github.com/openssl/openssl/issues/17451
On further debugging with Openssl 3.0, observed that
BIO_CTRL_SET_KTLS command is not received in sock_ctrl() function. so ktls_start() is never called.

> Regards
> Gaurav Jain