[openssl-users] Searching for a memory leak in my OpenSSL usage

[Hiesgen, Raphael]

Hello,

I got an application that establishes an TLS connection for communication. While the communication works, I run into a memory leak that originates from CRYPTO_malloc. I tried to search for proper OpenSSL shutdown and related issues, but my tries did not affect the leak. Here is the ASAN output:

```
==2618==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 672 byte(s) in 12 object(s) allocated from:
    #0 0x7f14a2c9bec0 in __interceptor_malloc (/usr/lib/x86_64-linux-gnu/libasan.so.3+0xc6ec0)
    #1 0x7f149fa4ce87 in CRYPTO_malloc (/lib/x86_64-linux-gnu/libcrypto.so.1.0.0+0x62e87)

Indirect leak of 15024 byte(s) in 228 object(s) allocated from:
    #0 0x7f14a2c9bec0 in __interceptor_malloc (/usr/lib/x86_64-linux-gnu/libasan.so.3+0xc6ec0)
    #1 0x7f149fa4ce87 in CRYPTO_malloc (/lib/x86_64-linux-gnu/libcrypto.so.1.0.0+0x62e87)

SUMMARY: AddressSanitizer: 15696 byte(s) leaked in 240 allocation(s).
```


My test establishes a connection between two threads handled by the same process. Both keep their separate SSL context. Just to be sure, that this is not a multithreading problem I tried passing locking and id functions to OpenSSL but that did change anything.

Here are the steps to reproduce the problem when you have OpenSSL installed via the system packet manager. If you want to use a specific SSL installation, the configure script accepts a '--with-openssl=PATH' argument.

```
git clone https://github.com/actor-framework/actor-framework.git
cd actor-framework
git checkout develop
./configure --build-type=debug --with-address-sanitizer
make
./build/bin/caf-test -s openssl_dynamic_remote_actor
```


The SSL session is created on top of an already established TCP connection using non-blocking socket.

For initialization [1],  we call create_ssl_context and SSL_new. This seems to be enough (?) and using Wireshark I can observe the handshake and so on. To clean up [2], we simply call SSL_free and SSL_CTX_free. Searching online, there are really many function for cleanup of different SSL parts, but that did not seem to do anything. Here are the functions, I tired (just to be sure, I tried them all in different configurations):

```
FIPS_mode_set(0);
EVP_cleanup();
SSL_COMP_free_compression_methods();
COMP_zlib_cleanup();
ERR_remove_state(0);
ERR_remove_thread_state(NULL);
CONF_modules_free();
CONF_modules_unload(1);
ERR_free_strings();
CRYPTO_cleanup_all_ex_data();
```


The multi threading additions are not in the branch, but I basically followed the libcurl example [3] but with a bit more C++. My host system is an Ubuntu 17.04 with the '4.10.0-35-generic' kernel, GCC ‘6.3.0 (20170406)' and 'OpenSSL 1.0.2g  1 Mar 2016'.

Thank you for your time!

Best Regards
Raphael


[1] https://github.com/actor-framework/actor-framework/blob/develop/libcaf_openssl/src/session.cpp#L56
[2] https://github.com/actor-framework/actor-framework/blob/develop/libcaf_openssl/src/session.cpp#L67
[3] https://curl.haxx.se/libcurl/c/opensslthreadlock.html