SSL_TLSEXT_ERR_NOACK not working as expected

[Tal Dery]

Hi Matt,
Yes, I want to abort the session without notifying the client.
SSL_TLSEXT_ERR_ALERT_FATAL sent " 15 03 03 00 02 02 00" to client.
What can I do to not send the message?

Thanks


-----Original Message-----
From: Matt Caswell <matt at openssl.org> 
Sent: Thursday, 10 March 2022 12:54
To: Tal Dery <tal at awrora.io>; openssl-users at openssl.org
Subject: Re: SSL_TLSEXT_ERR_NOACK not working as expected



On 10/03/2022 10:26, Tal Dery wrote:
> Hi,
> 
> I am implementing an SSL server.
> 
> Using SSL_CTX_set_tlsext_servername_callback I'm checking the SNI.
> 
> When SNI meets my requirements (for example does not contain offensive
> words) I allow the handshake by returning SSL_TLSEXT_ERR_OK.
> 
> When there is an offensive word, I do not want to send Server Hello 
> message. I try to do this by returning SSL_TLSEXT_ERR_NOACK.

If you don't want the ServerHello to be sent then you are aborting the handshake. In that case you should return SSL_TLSEXT_ERR_ALERT_FATAL. By comparison SSL_TLSEXT_ERR_NOACK is a non-fatal return code. The SNI request is not acknowledged by the server (i.e. it acts the same way as if SNI was not configured on the server at all), but no alerts are sent so the handshake proceeds as normal.

Matt



> 
> For some reason, the server is still sending the message, and I wonder why?
> 
> SSL_TLSEXT_ERR_ALERT_WARNING works as expected.
> 
> I'm using OpenSSL 1.1.1f and Wireshark to verify what I say.
> 
> Thanks
>