OPENSSL_init_crypto with OPENSSL_INIT_NO_ATEXIT issue

[Dan Heinz]

On 09/08/2019 14:33, Dan Heinz wrote:
>> I have a static library using OpenSSL (built as static library with 
>> the no-pinshared parameter in the configuration) that is then included 
>> in a DLL that gets loaded and unloaded many times by the calling 
>> application.  Now that the code is in 1.1.1c to allow me to manually 
>> shutdown the OpenSSL library, I've run into an issue when loading and 
>> unloading my DLL multiple times from the calling application.  I am testing this on Windows 10.  A summary of what I am doing:
>> 
>>  1. Calling executable calls LoadLibrary to load the DLL containing the static
>>     library that uses OpenSSL.
>>  2. Calling executable calls a function of the now loaded DLL.
>>  3. The DLL function calls a function in the static library to initialize
>>     OpenSSL using: openssl_result = OPENSSL_init_crypto(OPENSSL_INIT_NO_ATEXIT,
>>     NULL);
>>  4. The DLL function calls a function in the static library to decrypt some data
>>     using the RSA_private_decrypt OpenSSL API.
>>  5. The DLL function calls a function in the static library to shutdown OpenSSL
>>     using: OPENSSL_cleanup(); (required)  and CRYPTO_cleanup_all_ex_data(); (not
>>     sure if this is needed).
>>  6. The calling executable uses FreeLibrary to unload the DLL.
>>  7. The calling executable goes to step 1 to  repeat the process.
>> 
>> Iterating through the above steps always fails on iteration 1077.  
>> OpenSLL reports the error:
>> error:4088003:rsa routines:RSA_setup_binding:BN lib
>> 
>>  
>> 
>> After some debugging, I see that the failure happens in the function 
>> CRYPTO_THREAD_init_local in threads_win.c.  *key = TlsAlloc(); fails 
>> with the error TLS_OUT_OF_INDEXES.
>> 
>> Is something not being freed? Is there something additional I need to 
>> do at shutdown?

>For every call of CRYPTO_THREAD_init_local() while the library is running you should see a matching call to CRYPTO_THREAD_cleanup_local() to cleanup the index (as a result of an OPENSSL_cleanup() call). If you don't see that then an >index is leaking.


Thanks for the guidance, Matt.  I added some code to output the counts of the calls to TlsAlloc and TlsFree.  I see 4 allocations and 4 matching deallocations for each iteration of my tests.  
The allocations/deallocations are for:

destructor_key.value in OPENSSL_cleanup()
private_drbg    (from CRYPTO_THREAD_init_local(&private_drbg, NULL))
public_drbg     (from CRYPTO_THREAD_init_local(&public_drbg, NULL))
err_thread_local    (fromCRYPTO_THREAD_init_local(&err_thread_local, NULL))

On the 1076th test iteration, I get the error (TLS_OUT_OF_INDEXES) from TlsAlloc.

I checked for any errors from TlsFree during the tests and it always returns success.
One thing I did notice is that the index value (from this line: *key = TlsAlloc()) is always incremented by 1 after each test iteration.  The destructor_key.value always stayed the same number (7), but the other three allocations always incremented the value of the key variable.  By the time my test reaches the 1076th iteration, the value of key is just under 2000.  For example the indices from TlsAlloc might be 10,11, and 12 and the next iteration they are 11, 12, 13.  MS documentation states "The maximum number of indexes per process is 1,088"  which seems to coincide with the number of my test iterations.  None of my code uses thread local storage.  Any ideas?  

Here's some output from my tests (note the key values keep incrementing).
Iteration: 1    ***********************************************
Key value: 7
TlsAlloc called.  AllocationCount: 1
Key value: 9
TlsAlloc called.  AllocationCount: 2
Key value: 10
TlsAlloc called.  AllocationCount: 3
Key value: 11
TlsAlloc called.  AllocationCount: 4
TlsFree called.   AllocationCount: 3
TlsFree called.   AllocationCount: 2
TlsFree called.   AllocationCount: 1
TlsFree called.   AllocationCount: 0
Iteration: 2    ***********************************************
Key value: 7
TlsAlloc called.  AllocationCount: 1
Key value: 10
TlsAlloc called.  AllocationCount: 2
Key value: 11
TlsAlloc called.  AllocationCount: 3
Key value: 12
TlsAlloc called.  AllocationCount: 4
TlsFree called.   AllocationCount: 3
TlsFree called.   AllocationCount: 2
TlsFree called.   AllocationCount: 1
TlsFree called.   AllocationCount: 0 

.....

Iteration: 1076    ***********************************************
Key value: 7
TlsAlloc called.  AllocationCount: 1
Key value: 1086
TlsAlloc called.  AllocationCount: 2
Key value: 1087
TlsAlloc called.  AllocationCount: 3
Key value: 4294967295 (this is TLS_OUT_OF_INDEXES)