SSL_CTX_set_alpn_select_cb and Other OpenSSL API ALPN Functions + Their Callbacks

[Matt Caswell]

On 29/07/2022 17:21, Angus Robertson - Magenta Systems Ltd wrote:
>> I don't understand how to write the callback functions some of
>> the OpenSSL ALPN functions expect, and the manual really isn't
>> helping there either, so I'd like some help.
> 
> Use SSL_CTX_set_client_hello_cb to set a SSL_client_hello_cb_fn
> function, which you can parse to get TLSEXT_TYPE_server_name and
> TLSEXT_TYPE_application_layer_protocol_negotiation, and everything else
> sent in the Client Hello (if you need it) like SSL versions and ciphers
> supported.
> 
> Within this callback you can change SSL_CTX depending on SNI and ALPN.
> 
> 
> Ignore the SNI and ALPN callbacks.  client_hello_cb was only added in
> 1.1.1 so is often missing from old examples, FAQs and manuals.

While this may be reasonable advice for SNI, I'm not sure that this is 
correct for ALPN. I don't think it is actually possible to set the 
selected ALPN *without* using the ALPN callback. At least I can't see a way.

A useful addition to OpenSSL might be a new API to set the selected ALPN 
directly which could be called from a client_hello_cb.

There's an example of an alpn selection callback here:

https://github.com/openssl/openssl/blob/72a85c17aae602e881c917c3f6e93bd7f7260093/apps/s_server.c#L643-L680

https://github.com/openssl/openssl/blob/72a85c17aae602e881c917c3f6e93bd7f7260093/apps/s_server.c#L1786-L1791

https://github.com/openssl/openssl/blob/72a85c17aae602e881c917c3f6e93bd7f7260093/apps/s_server.c#L2048-L2049


Matt