[openssl-dev] [BUG] Different heap implementations in engine_pkcs11.dll and OpenSSL cause crash at EVP_PKEY_free(cpk->privatekey)

[Paweł Witas]

Thanks for fast response.

Yes, engine_pkcs11.dll and libp11.dll are statically linked with OpenSSL
(compiled by mingw gcc), but affected Libcurl.dll (or Curl.exe or
Openssl.exe) compiled by MS Visual Studio, uses dynamically linked OpenSSL,
that loads configuration file, that asks OpenSSL to load this engine.
As a result, engine_pkcs11.dll allocates memory on its private heap by
using its own statically linked OpenSSL, that uses different heap
implementation than OpenSSL used by some other programs.


On Thu, Dec 10, 2015 at 12:14 PM, Richard Levitte <levitte at openssl.org>
wrote:

> Hey Paweł,
>
> In message <
> CAPv+SJGWXnbkJPd4ykYOTMEAEib-wcP5DETbeZM2XnSV9id4Vg at mail.gmail.com> on
> Thu, 10 Dec 2015 10:20:02 +0100, Paweł Witas <pw178860 at gmail.com> said:
>
> pw178860> I'm working on implementing PKCS#11 encrypted communication on
> Windows
> pw178860> platform.
> pw178860> This crash occurs on Windows Vista and above when
> engine_pkcs11.dll is
> pw178860> compiled by mingw toolchain and OpenSSL is compiled by Visual
> Studio
> pw178860> 2012.
> pw178860> It does not occur on Windows XP or when both engine_pkcs11.dll
> and
> pw178860> OpenSSL are compiled by mingw toolchain.
> pw178860>
> pw178860> The cause of this crash are different and incompatible
> implementations
> pw178860> of memory allocators in engine_pkcs11.dll (from Windows kernel's
> pw178860> msvcrt.dll) and OpenSSL (from VS212 msvcr110.dll).
> pw178860> The private key is allocated by engine_pkcs11.dll on its private
> heap
> pw178860> via callback from OpenSSL, but freed by the OpenSSL library
> itself.
>
> Here's a detail, IMPLEMENT_DYNAMIC_BIND_FN, passes down the memory
> allocation callbacks to the OpenSSL that the engine uses (be it
> statically linked or not).  This means that memory allocated with
> OPENSSL_malloc and friends in engine_pkcs11.dll *should* be allocated
> in OpenSSL library heap rather than engine_pkcs11.dll private heap.
> However, if anything returned through EVP_PKEY* returns (I suspect the
> issues come up around keys) is allocated using something other than
> OPENSSL allocation calls (such as malloc(), calloc()...), then yes,
> you do get in trouble.  I just spent a bit of time eyeing through
> engine_pkcs11 and libp11 and couldn't immediately find a culprit of
> that sort, but ...
>
> pw178860> This is troublesome, because I can compile OpenSSL by mingw for
> my
> pw178860> clients and put it at the beginning of the PATH,, but I can't
> replace
> pw178860> OpenSSL statically linked with third party products, i.e.
> Symantec
> pw178860> Antivirus LicenseMan.dll, which causes antivirus crash, because
> it
> pw178860> loads my openssl.cnf with engine_pkcs11.dll configured and tries
> to
> pw178860> use it (why?).
>
> I have no idea what LicenseMan.dll is or why it uses OpenSSL.
> However, if it does, it may very well call OPENSSL_config or
> CONF_modules_load_file, which will load whatever configuration file
> that's given through the OPENSSL_CONF environment variable.  It might
> be that they need to use a different default section (the default is
> "openssl_conf"); that might be something to report back to Symantec.
>
> pw178860> I solved this problem by replacing references to environment
> variable
> pw178860> "OPENSSL_CONF" by "OPENSSL_KONF" in my compilation of OpenSSL,
> but the
> pw178860> real solution would be allowing the engine_pkcs11.dll library to
> pw178860> deallocate its own keys by the deallocating callback from
> OpenSSL.
> pw178860> It will require modifications to both OpenSSL and
> engine_pkcs11.dll
> pw178860> library.
>
> I agree with those thoughts.  That will be for a future major OpenSSL
> version, I cannot say which one for the moment.
>
> --
> Richard Levitte         levitte at openssl.org
> OpenSSL Project         http://www.openssl.org/~levitte/
> _______________________________________________
> openssl-dev mailing list
> To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mta.openssl.org/pipermail/openssl-dev/attachments/20151210/9acc6f33/attachment-0001.html>