[openssl-dev] [openssl.org #4559] bug: CRYPTO_set_mem_functions() Doesn't Work in Version 1.0.1b

[Dan Kegel via RT]

The commit
  From: "Dr. Stephen Henson" <steve at openssl.org>
  Date: Fri, 1 Apr 2011 15:46:03 +0000
  Subject: [PATCH] Add additional OPENSSL_init() handling add dummy call to
  (hopefully) ensure OPENSSL_init() is always linked into an application.
  https://github.com/openssl/openssl/commit/c4acfb1fd049f52fb074b103be01cab5cf5c04f8

seems to have broken CRYPTO_set_mem_functions.  After this commit,
CRYPTO_set_mem_functions() calls OPENSSL_init(), which calls
CRYPTO_malloc(), which sets `allow_customize` to 0, which is then
checked by CRYPTO_set_mem_functions(), which then returns without
doing anything.

See also earlier reports
  http://openssl.6102.n7.nabble.com/CRYPTO-set-mem-functions-Doesn-t-Work-in-Version-1-0-1b-td46745.html
  http://bugs.python.org/msg191610

The death test program

#include <stdio.h>
#include <stdlib.h>
#include <openssl/ssl.h>
void * my_alloc(size_t n) { abort(); }
void my_free(void *p) { abort(); }
void * my_realloc(void *p, size_t n) { abort(); }
int main(int argc, const char **argv)
{
  const SSL_METHOD *method;
  SSL_CTX *ctx;
  CRYPTO_set_mem_functions(my_alloc, my_realloc, my_free);
  SSL_library_init();
  method = SSLv23_client_method();
  ctx = SSL_CTX_new(method);
  printf("Got ctx %p\n", ctx);
  return 0;
}

aborts as expected on Ubuntu 10.04 with openssl0.9.8k, but blithely
prints a ctx with openssl 1.0.*.

The bug is probably not present in openssl-1.1.0-pre5, as the
offending code was removed by
https://github.com/openssl/openssl/commit/bbd86bf5424a611cb6b77a3a17fc522931c4dcb8
but a fix for 1.0.0 and 1.0.1 would be much appreciated.


-- 
Ticket here: http://rt.openssl.org/Ticket/Display.html?id=4559
Please log in as guest with password guest if prompted