[openssl-dev] [openssl.org #3905] Bug report: segfault while cleaning up in libgost

[bug-reporting0000@cneufeld.ca via RT]

On Mon, 15 Jun 2015 03:29:12 +0000, Rich Salz via RT <rt at openssl.org> said:

> It's strange that CRYPTO_malloc ends up calling ISC-malloc, but CRYPTO_free
> does not. This is strange, but really hard to see how this is an openssl issue.

OK, I've been poking at this a bit more.  The ISC bind's host program is
calling CRYPTO_set_mem_functions, and registering its malloc/free
analogues.  That's going fine.  They are used when gost_set_default_param()
is called during setup.  malloc_func, free_func, and so on in crypto/mem.c
are set, and everything looks good.  CRYPTO_set_mem_functions itself lies
in the mmap-ed range of /usr/lib/libcrypto.so, as expected.

Then, the program runs, and it goes to shut down.  I'll note that it seems
to signal itself for shutdown by sending itself a catchable signal
(SIGTERM), but execution continues.  Eventually, it gets to
gost_param_free(), and if you inspect malloc_func and free_func here,
they're still correct, set to the ISC analogues.  But, when you descend
into CRYPTO_free, suddenly they're set to glibc malloc/free again.  This
confused me for a while because hardware watchpoints didn't spot the
switch.  But it turns out, they're not in the same mapped region.  The
malloc_func and free_func seen now are in the /usr/lib/engines/libgost.so
mapping, not the /usr/lib/libcrypto.so mapping.  Turns out libgost has its
own copy of those symbols, at different addresses, and those ones haven't
been modified by CRYPTO_set_mem_functions.  Inspection with "nm" shows
this.  In fact, the CRYPTO_free() that we're calling at this point isn't
the one mapped into libcrypto, but is libgost's own separate copy, and it's
using the unmodified memory hooks, since it never saw
CRYPTO_set_mem_functions.

It appears to me that mem.c is linked into the binary through two different
routes, and both are being used at various times in the execution of the
program.  The 'host' program does not spawn threads, everything is
happening in the same stack.

I'll note that libgost.so is dynamically linked against libcrypto.so, but
that's not where the second incarnation appears, 'nm' shows the symbols to
be linked into the libgost.so library directly (the malloc_func and
malloc_free symbols appear with 'd', the CRYPTO_free appears with 'T', so
they're all present as compiled symbols in libgost.so).


So, this may be a build issue, though I haven't done anything particularly
unusual in my configuration:
./config zlib shared threads no-hw no-krb5 --prefix=/usr

The link line in make when building libgost.so doesn't include mem.o in the
objects list, but I'll do some more checking later.


-- 
 Christopher Neufeld
 Home page:  http://www.cneufeld.ca/neufeld
 "Don't edit reality for the sake of simplicity"