[EXT] Re: KTLS with openssl 3.0 fail with error ENOTCONN(Transport endpoint is not connected)

[John Baldwin]

On 1/6/22 5:58 AM, Gaurav Jain wrote:
> Hi
> 
>> -----Original Message-----
>> From: John Baldwin <jhb at FreeBSD.org>
>> Sent: Thursday, January 6, 2022 12:26 AM
>> To: Gaurav Jain <gaurav.jain at nxp.com>; borisp at mellanox.com; openssl-
>> users at openssl.org
>> Cc: Varun Sethi <V.Sethi at nxp.com>; Pankaj Gupta <pankaj.gupta at nxp.com>
>> Subject: [EXT] Re: KTLS with openssl 3.0 fail with error ENOTCONN(Transport
>> endpoint is not connected)
>>
>> Caution: EXT Email
>>
>> On 1/4/22 11:49 PM, Gaurav Jain wrote:
>>> Hello Boris/John
>>>
>>> I am from NXP and currently working on enabling KTLS on NXP platforms via
>> openssl.
>>> I see that you enabled KTLS support in openssl
>> 3.0(https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww
>> .openssl.org%2Fnews%2Fchangelog.html%23openssl-
>> 30&data=04%7C01%7Cgaurav.jain%40nxp.com%7Ce87da43a5488475b2aa
>> d08d9d07d05b0%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C1%7C63777
>> 0057654781203%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQ
>> IjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=vZa0aCu
>> D%2FzrXB0vv23DZiOWSVichep42YLqA4a1JeXY%3D&reserved=0).
>>>
>>> when I configure openssl 3.0 or 3.1.0 with enable-ktls and and try to run the
>> s_server, s_client application.
>>> I observe that connection is successfully established - but it didn't use KTLS.
>>>
>>> Then I added additional log in kernel(file net/tls/tls_main.c) and see
>>> that kernel is returning error -ENOTCONN when (sk->sk_state !=
>>> TCP_ESTABLISHED) in function static int tls_init(struct sock *sk)
>>
>> To be clear, I have worked on KTLS support for FreeBSD, not for Linux.
>>
>> However, I think the error you are seeing is a red herring.  I think you are seeing
>> the setsockopt() call from ktls_enable() fail because it is invoked on the listen
>> socket since ktls_enable() is called when sockets are created by libssl.
>>
>> For KTLS to work on the server side on Linux what you need to find out is when
>> ktls_enable() is invoked on the socket returned by accept() and why that is failing.
>>
> 
> Thanks John for your input.
> Ktls_enable() after accept() is successful on server side.
> I added debug logs, ktls_start() is failing with error Invalid argument.
> 
> Logs:
> openssl s_server -ktls -key rsa.key -cert server.pem -accept 443
> 
> Using default temp DH parameters
> 
> ACCEPT
> 
> ktls_enable setsockopt success, ret = 0
> 
> ktls_enable() = 1
> 
> 
> fd = 4, is_tx = 0, tls_crypto_info_len = 1872610871009456445
> 
> ktls_start setsockopt failed, 22, Invalid argument
> 
> fd = 4, is_tx = 2, tls_crypto_info_len = 8329596950154514032
> 
> ktls_start setsockopt failed, 22, Invalid argument

You'd have to add traces to see why this is failing I think.  I know on FreeBSD
we will fail the equivalent setsockopt if the ciphersuite isn't supported (e.g.
some cipher suites we only support if the socket is routed over a NIC with offload
support, and there can also be system tunables to disable certain cipher suites).
I don't quite know what that is like on the Linux side though.

-- 
John Baldwin