OPENSSL_init_crypto with OPENSSL_INIT_NO_ATEXIT issue

[Matt Caswell]

On 09/08/2019 14:33, Dan Heinz wrote:
> I have a static library using OpenSSL (built as static library with the
> no-pinshared parameter in the configuration) that is then included in a DLL that
> gets loaded and unloaded many times by the calling application.  Now that the
> code is in 1.1.1c to allow me to manually shutdown the OpenSSL library, I’ve run
> into an issue when loading and unloading my DLL multiple times from the calling
> application.  I am testing this on Windows 10.  A summary of what I am doing:
> 
>  1. Calling executable calls LoadLibrary to load the DLL containing the static
>     library that uses OpenSSL.
>  2. Calling executable calls a function of the now loaded DLL.
>  3. The DLL function calls a function in the static library to initialize
>     OpenSSL using: openssl_result = OPENSSL_init_crypto(OPENSSL_INIT_NO_ATEXIT,
>     NULL);
>  4. The DLL function calls a function in the static library to decrypt some data
>     using the RSA_private_decrypt OpenSSL API.
>  5. The DLL function calls a function in the static library to shutdown OpenSSL
>     using: OPENSSL_cleanup(); (required)  and CRYPTO_cleanup_all_ex_data(); (not
>     sure if this is needed).
>  6. The calling executable uses FreeLibrary to unload the DLL.
>  7. The calling executable goes to step 1 to  repeat the process.
> 
> Iterating through the above steps always fails on iteration 1077.  OpenSLL
> reports the error:
> error:4088003:rsa routines:RSA_setup_binding:BN lib
> 
>  
> 
> After some debugging, I see that the failure happens in the function
> CRYPTO_THREAD_init_local in threads_win.c.  *key = TlsAlloc(); fails with the
> error TLS_OUT_OF_INDEXES.
> 
> Is something not being freed? Is there something additional I need to do at
> shutdown? 

For every call of CRYPTO_THREAD_init_local() while the library is running you
should see a matching call to CRYPTO_THREAD_cleanup_local() to cleanup the index
(as a result of an OPENSSL_cleanup() call). If you don't see that then an index
is leaking.

Matt