Need help understanding how the custom extension interacts with the SSL pointer, if at all

[Matt Caswell]

On 19/10/2023 14:51, Xavier Marchal wrote:
> Hello,
> 
> In the context of a research project I need to add some extensions to 
> the ClientHello during TLS handshake but I don't understand well some 
> concepts of the custom extensions.
> 
> I can successfully send custom extensions between my client and server 
> thanks to the SSL_CTX_add_custom_ext function but I have a hard time to 
> use these values.
> 
> I currently define them like this on both sides: 
> SSL_CTX_add_custom_ext(ssl_ctx, 101, SSL_EXT_CLIENT_HELLO, addScalar, 
> freeScalar, NULL, parseScalar, NULL);
> 
> What I want to do is to store the value of the extension in a structure 
> linked with with each SSL sessions pointer I have but the callbacks are 
> set at the context level so I don't think I can give pointers to my 
> structures easily as they do no exist yet when the custom extension is 
> defined.

It's a bit unclear from your description exactly what you are trying to 
do. But IIUC you want to associate custom data with the SSL object. Many 
OpenSSL objects (including the SSL object) support the "ex_data" 
interface which enables you to store and retrieve custom data associated 
with the object.

See in particular:

https://www.openssl.org/docs/man3.1/man3/CRYPTO_get_ex_new_index.html

The SSL_get_app_data() and SSL_set_app_data() convenience macros wrap 
"ex_data" to give a simplified interface:

https://www.openssl.org/docs/man3.1/man3/SSL_get_app_data.html

E.g. call SSL_set_app_data() to associate a custom pointer with an SSL, 
and SSL_get_app_data() to retrieve it again later.

Matt


> 
> I think it may be possible to keep a global map with SSL session 
> pointers as keys but I am not sure it is the way to do.
> 
> Or maybe I can do a 1:1 with only a session per context but it looks 
> suboptimal.
> 
> In the same way, is it possible for a SSL client to set a specific value 
> for a custom extension if it only has access to a SSL pointer? (in my 
> case it would be better if I have only one SSL_CTX for all SSL clients)
> 
> Is what I'm trying to do feasible?
> 
> Regards,
> 
> Xavier Marchal