[openssl-dev] [BUG] Different heap implementations in engine_pkcs11.dll and OpenSSL cause crash at EVP_PKEY_free(cpk->privatekey)

[Paweł Witas]

Hello

I'm working on implementing PKCS#11 encrypted communication on Windows
platform.
This crash occurs on Windows Vista and above when engine_pkcs11.dll is
compiled by mingw toolchain and OpenSSL is compiled by Visual Studio 2012.
It does not occur on Windows XP or when both engine_pkcs11.dll and OpenSSL
are compiled by mingw toolchain.

The cause of this crash are different and incompatible implementations of
memory allocators in engine_pkcs11.dll (from Windows kernel's msvcrt.dll)
and OpenSSL (from VS212 msvcr110.dll).
The private key is allocated by engine_pkcs11.dll on its private heap via
callback from OpenSSL, but freed by the OpenSSL library itself.

This is troublesome, because I can compile OpenSSL by mingw for my clients
and put it at the beginning of the PATH,, but I can't replace OpenSSL
statically linked with third party products, i.e. Symantec Antivirus
LicenseMan.dll, which causes antivirus crash, because it loads my
openssl.cnf with engine_pkcs11.dll configured and tries to use it (why?).

I solved this problem by replacing references to environment variable
"OPENSSL_CONF" by "OPENSSL_KONF" in my compilation of OpenSSL, but the real
solution would be allowing the engine_pkcs11.dll library to deallocate its
own keys by the deallocating callback from OpenSSL.
It will require modifications to both OpenSSL and engine_pkcs11.dll library.

Best regards
Pawel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mta.openssl.org/pipermail/openssl-dev/attachments/20151210/5ec77f37/attachment.html>