SSL_TLSEXT_ERR_NOACK not working as expected

[Matt Caswell]

On 10/03/2022 10:26, Tal Dery wrote:
> Hi,
> 
> I am implementing an SSL server.
> 
> Using SSL_CTX_set_tlsext_servername_callback I'm checking the SNI.
> 
> When SNI meets my requirements (for example does not contain offensive 
> words) I allow the handshake by returning SSL_TLSEXT_ERR_OK.
> 
> When there is an offensive word, I do not want to send Server Hello 
> message. I try to do this by returning SSL_TLSEXT_ERR_NOACK.

If you don't want the ServerHello to be sent then you are aborting the 
handshake. In that case you should return SSL_TLSEXT_ERR_ALERT_FATAL. By 
comparison SSL_TLSEXT_ERR_NOACK is a non-fatal return code. The SNI 
request is not acknowledged by the server (i.e. it acts the same way as 
if SNI was not configured on the server at all), but no alerts are sent 
so the handshake proceeds as normal.

Matt



> 
> For some reason, the server is still sending the message, and I wonder why?
> 
> SSL_TLSEXT_ERR_ALERT_WARNING works as expected.
> 
> I'm using OpenSSL 1.1.1f and Wireshark to verify what I say.
> 
> Thanks
>