OPENSSL_init_crypto with OPENSSL_INIT_NO_ATEXIT issue

[Matt Caswell]

On 13/08/2019 19:16, Dan Heinz wrote:
> On 09/08/2019 14:33, Dan Heinz wrote:
>>> I have a static library using OpenSSL (built as static library with the
>>> no-pinshared parameter in the configuration) that is then included in a
>>> DLL that gets loaded and unloaded many times by the calling application.
>>> Now that the code is in 1.1.1c to allow me to manually shutdown the
>>> OpenSSL library, I've run into an issue when loading and unloading my DLL
>>> multiple times from the calling application.  I am testing this on
>>> Windows 10.  A summary of what I am doing:
>>> 
>>> 1. Calling executable calls LoadLibrary to load the DLL containing the
>>> static library that uses OpenSSL. 2. Calling executable calls a function
>>> of the now loaded DLL. 3. The DLL function calls a function in the static
>>> library to initialize OpenSSL using: openssl_result =
>>> OPENSSL_init_crypto(OPENSSL_INIT_NO_ATEXIT, NULL); 4. The DLL function
>>> calls a function in the static library to decrypt some data using the
>>> RSA_private_decrypt OpenSSL API. 5. The DLL function calls a function in
>>> the static library to shutdown OpenSSL using: OPENSSL_cleanup();
>>> (required)  and CRYPTO_cleanup_all_ex_data(); (not sure if this is
>>> needed). 6. The calling executable uses FreeLibrary to unload the DLL. 7.
>>> The calling executable goes to step 1 to  repeat the process.
>>> 
>>> Iterating through the above steps always fails on iteration 1077. OpenSLL
>>> reports the error: error:4088003:rsa routines:RSA_setup_binding:BN lib
>>> 
>>> 
>>> 
>>> After some debugging, I see that the failure happens in the function 
>>> CRYPTO_THREAD_init_local in threads_win.c.  *key = TlsAlloc(); fails with
>>> the error TLS_OUT_OF_INDEXES.
>>> 
>>> Is something not being freed? Is there something additional I need to do
>>> at shutdown?
> 
>> For every call of CRYPTO_THREAD_init_local() while the library is running
>> you should see a matching call to CRYPTO_THREAD_cleanup_local() to cleanup
>> the index (as a result of an OPENSSL_cleanup() call). If you don't see that
>> then an >index is leaking.
> 
> 
> Thanks for the guidance, Matt.  I added some code to output the counts of the
> calls to TlsAlloc and TlsFree.  I see 4 allocations and 4 matching
> deallocations for each iteration of my tests. The allocations/deallocations
> are for:
> 
> destructor_key.value in OPENSSL_cleanup() private_drbg    (from
> CRYPTO_THREAD_init_local(&private_drbg, NULL)) public_drbg     (from
> CRYPTO_THREAD_init_local(&public_drbg, NULL)) err_thread_local
> (fromCRYPTO_THREAD_init_local(&err_thread_local, NULL))
> 
> On the 1076th test iteration, I get the error (TLS_OUT_OF_INDEXES) from
> TlsAlloc.
> 
> I checked for any errors from TlsFree during the tests and it always returns
> success. One thing I did notice is that the index value (from this line: *key
> = TlsAlloc()) is always incremented by 1 after each test iteration.  The
> destructor_key.value always stayed the same number (7), but the other three
> allocations always incremented the value of the key variable.

That is quite strange behaviour. I can't think of a good reason why that
particular index should always be the same while the others increment.

I suppose the 4 deallocation calls that you see are deallocating the correct
index? And that the TlsFree calls actually succeed?

I'm not really a Windows programmer, so perhaps others who have more knowledge
of Windows than I do can make some suggestions.

Matt


>  By the time my
> test reaches the 1076th iteration, the value of key is just under 2000.  For
> example the indices from TlsAlloc might be 10,11, and 12 and the next
> iteration they are 11, 12, 13.  MS documentation states "The maximum number
> of indexes per process is 1,088"  which seems to coincide with the number of
> my test iterations.  None of my code uses thread local storage.  Any ideas?
> 
> 
> Here's some output from my tests (note the key values keep incrementing). 
> Iteration: 1    *********************************************** Key value: 7 
> TlsAlloc called.  AllocationCount: 1 Key value: 9 TlsAlloc called.
> AllocationCount: 2 Key value: 10 TlsAlloc called.  AllocationCount: 3 Key
> value: 11 TlsAlloc called.  AllocationCount: 4 TlsFree called.
> AllocationCount: 3 TlsFree called.   AllocationCount: 2 TlsFree called.
> AllocationCount: 1 TlsFree called.   AllocationCount: 0 Iteration: 2
> *********************************************** Key value: 7 TlsAlloc called.
> AllocationCount: 1 Key value: 10 TlsAlloc called.  AllocationCount: 2 Key
> value: 11 TlsAlloc called.  AllocationCount: 3 Key value: 12 TlsAlloc called.
> AllocationCount: 4 TlsFree called.   AllocationCount: 3 TlsFree called.
> AllocationCount: 2 TlsFree called.   AllocationCount: 1 TlsFree called.
> AllocationCount: 0
> 
> .....
> 
> Iteration: 1076    *********************************************** Key value:
> 7 TlsAlloc called.  AllocationCount: 1 Key value: 1086 TlsAlloc called.
> AllocationCount: 2 Key value: 1087 TlsAlloc called.  AllocationCount: 3 Key
> value: 4294967295 (this is TLS_OUT_OF_INDEXES)
> 
> 
>