[openssl-dev] [BUG] Different heap implementations in engine_pkcs11.dll and OpenSSL cause crash at EVP_PKEY_free(cpk->privatekey)

[Richard Levitte]

Hey Paweł,

In message <CAPv+SJGWXnbkJPd4ykYOTMEAEib-wcP5DETbeZM2XnSV9id4Vg at mail.gmail.com> on Thu, 10 Dec 2015 10:20:02 +0100, Paweł Witas <pw178860 at gmail.com> said:

pw178860> I'm working on implementing PKCS#11 encrypted communication on Windows
pw178860> platform.
pw178860> This crash occurs on Windows Vista and above when engine_pkcs11.dll is
pw178860> compiled by mingw toolchain and OpenSSL is compiled by Visual Studio
pw178860> 2012.
pw178860> It does not occur on Windows XP or when both engine_pkcs11.dll and
pw178860> OpenSSL are compiled by mingw toolchain.
pw178860> 
pw178860> The cause of this crash are different and incompatible implementations
pw178860> of memory allocators in engine_pkcs11.dll (from Windows kernel's
pw178860> msvcrt.dll) and OpenSSL (from VS212 msvcr110.dll).
pw178860> The private key is allocated by engine_pkcs11.dll on its private heap
pw178860> via callback from OpenSSL, but freed by the OpenSSL library itself.

Here's a detail, IMPLEMENT_DYNAMIC_BIND_FN, passes down the memory
allocation callbacks to the OpenSSL that the engine uses (be it
statically linked or not).  This means that memory allocated with
OPENSSL_malloc and friends in engine_pkcs11.dll *should* be allocated
in OpenSSL library heap rather than engine_pkcs11.dll private heap.
However, if anything returned through EVP_PKEY* returns (I suspect the
issues come up around keys) is allocated using something other than
OPENSSL allocation calls (such as malloc(), calloc()...), then yes,
you do get in trouble.  I just spent a bit of time eyeing through
engine_pkcs11 and libp11 and couldn't immediately find a culprit of
that sort, but ...

pw178860> This is troublesome, because I can compile OpenSSL by mingw for my
pw178860> clients and put it at the beginning of the PATH,, but I can't replace
pw178860> OpenSSL statically linked with third party products, i.e. Symantec
pw178860> Antivirus LicenseMan.dll, which causes antivirus crash, because it
pw178860> loads my openssl.cnf with engine_pkcs11.dll configured and tries to
pw178860> use it (why?).

I have no idea what LicenseMan.dll is or why it uses OpenSSL.
However, if it does, it may very well call OPENSSL_config or
CONF_modules_load_file, which will load whatever configuration file
that's given through the OPENSSL_CONF environment variable.  It might
be that they need to use a different default section (the default is
"openssl_conf"); that might be something to report back to Symantec.

pw178860> I solved this problem by replacing references to environment variable
pw178860> "OPENSSL_CONF" by "OPENSSL_KONF" in my compilation of OpenSSL, but the
pw178860> real solution would be allowing the engine_pkcs11.dll library to
pw178860> deallocate its own keys by the deallocating callback from OpenSSL.
pw178860> It will require modifications to both OpenSSL and engine_pkcs11.dll
pw178860> library.

I agree with those thoughts.  That will be for a future major OpenSSL
version, I cannot say which one for the moment.

-- 
Richard Levitte         levitte at openssl.org
OpenSSL Project         http://www.openssl.org/~levitte/