[openssl-users] How to send alert in handshake?

[Matt Caswell]

On 27/06/18 12:41, Felipe Gasper wrote:
> RFC 3546, in describing the SNI extension, recommends that servers send a warning to clients that request an unknown server name. (Page 9)
> 
> I’d like to implement that warning .. could someone please point me to which API functions expose this ability?

In order to implement SNI you need an SNI callback which can be set with
this function:

 long SSL_CTX_set_tlsext_servername_callback(SSL_CTX *ctx,
                                   int (*cb)(SSL *, int *, void *));

It is (briefly) documented in 1.1.1 (but not earlier versions - although
it exists in those versions):

https://www.openssl.org/docs/man1.1.1/man3/SSL_CTX_set_tlsext_servername_callback.html

The documentation seems to be missing some vital information though -
like what the arguments to the callback mean and what the return value does!

The callback should return:

SSL_TLSEXT_ERR_OK, if it successfully processed the SNI
SSL_TLSEXT_ERR_ALERT_WARNING, to send a warning alert back
SSL_TLSEXT_ERR_ALERT_FATAL, to send a fatal alert back
SSL_TLSEXT_ERR_NOACK, to continue without acknowledging the SNI at all

The alert, if sent, will default to SSL_AD_UNRECOGNIZED_NAME but you can
fill in the second "int *" argument to the callback with some other
alert value if you wish.

>From the callback you can determine what the requested servername was by
calling SSL_get_servername().

Note though that RFC 3546 that you reference is obsolete. It was
obsoleted by RFC 4366, which itself was obsoleted by RFC 6066. That last
RFC has this to say about fatal vs warning alerts:

   If the server understood the ClientHello extension but
   does not recognize the server name, the server SHOULD take one of two
   actions: either abort the handshake by sending a fatal-level
   unrecognized_name(112) alert or continue the handshake.  It is NOT
   RECOMMENDED to send a warning-level unrecognized_name(112) alert,
   because the client's behavior in response to warning-level alerts is
   unpredictable.  If there is a mismatch between the server name used
   by the client application and the server name of the credential
   chosen by the server, this mismatch will become apparent when the
   client application performs the server endpoint identification, at
   which point the client application will have to decide whether to
   proceed with the communication.  TLS implementations are encouraged
   to make information available to application callers about warning-
   level alerts that were received or sent during a TLS handshake.  Such
   information can be useful for diagnostic purposes.


Matt