[openssl-dev] Core dump OpenSSL 1.1.0-pre5 during test (likely in 70-test_sslskewith0p.t)

[Rainer Jung]

I get a core dump during test execution for 1.1.0-pre5. Test is 
test/recipes/70-test_sslskewith0p.t, platform is Solaris 10 Sparc.

The crash happens in BN_num_bits() (crypto/bn/bn_lib.c:211), because it 
is called with a NULL argument, which is then dereferenced.

The call comes from EVP_PKEY_bits() (crypto/evp/p_lib.c:78). There the 
key has pkey->ameth->pkey_bits equals to dh_bits(), so it calls 
dh_bits(pkey), which calls BN_num_bits(pkey->pkey.dh->p). Looking at 
pkey->pkey.dh we see:

(gdb) print *pkey->pkey.dh
$6 = {pad = 0, version = 0, p = 0x0, g = 0x0, length = 0, pub_key = 0x0, 
priv_key = 0x0, flags = 1025, method_mont_p = 0x0, q = 0x0, j = 0x0, 
seed = 0x0, seedlen = 0,
   counter = 0x0, references = 1, ex_data = {sk = 0x0}, meth = 0x2b45ec, 
engine = 0x0, lock = 0x2e2c68}

so p is actually NULL.

Further print stack:

#2  0x00072e54 in ssl_print_tmp_key (out=out at entry=0x2dae68, 
s=s at entry=0x2e1ab0) at apps/s_cb.c:478
#3  0x00075894 in print_stuff (bio=0x2dae68, s=s at entry=0x2e1ab0, 
full=full at entry=1) at apps/s_client.c:2625
#4  0x000784c8 in s_client_main (argc=<optimized out>, argv=<optimized 
out>) at apps/s_client.c:2521

Verbose run output:

$ HARNESS_VERBOSE=yes TESTS=test_sslskewith0p make test
( cd test; \
   SRCTOP=../. \
   BLDTOP=../. \
   EXE_EXT= \
   OPENSSL_ENGINES=.././engines \
     /usr/bin/env perl .././test/run_tests.pl test_sslskewith0p )
../test/recipes/70-test_sslskewith0p.t ..
1..1
Proxy started on port 4453
engine "ossltest" set.
Using default temp DH parameters
ACCEPT
engine "ossltest" set.
Connection opened
Received client packet
Packet length = 102
Processing flight 0
  Record 1 (client -> server)
   Content type: HANDSHAKE
   Version: TLS1.0
   Length: 97
   Message type: ClientHello
   Message Length: 93
     Client Version:771
     Session ID Len:0
     Ciphersuite len:4
     Compression Method Len:1
     Extensions Len:48

Forwarded packet length = 102
Received server packet
Packet length = 474
Processing flight 1
  Record 1 (server -> client)
   Content type: HANDSHAKE
   Version: TLS1.2
   Length: 61
   Message type: ServerHello
   Message Length: 57
     Server Version:771
     Session ID Len:0
     Ciphersuite:52
     Compression Method:0
     Extensions Len:17
  Record 2 (server -> client)
   Content type: HANDSHAKE
   Version: TLS1.2
   Length: 394
   Message type: ServerKeyExchange
   Message Length: 390
  Record 3 (server -> client)
   Content type: HANDSHAKE
   Version: TLS1.2
   Length: 4
   Message type: ServerHelloDone
   Message Length: 0

Forwarded packet length = 347
1:error:1416D066:SSL routines:tls_process_key_exchange:bad dh 
value:ssl/statem/statem_clnt.c:1568:
Received client packet
Packet length = 7
Processing flight 2
  Record 1 (client -> server)
   Content type: ALERT
   Version: TLS1.2
   Length: 2

Forwarded packet length = 7
Connection closed
Waiting for server process to close: 14314
    0 items in the session cache
    0 client connects (SSL_connect())
    0 client renegotiates (SSL_connect())
    0 client connects that finished
    1 server accepts (SSL_accept())
    0 server renegotiates (SSL_accept())
    0 server accepts that finished
    0 session cache hits
    0 session cache misses
    0 session cache timeouts
    0 callback cache hits
    0 cache full overflows (128 allowed)
CONNECTION FAILURE
1:error:1409441A:SSL 
routines:ssl3_read_bytes:reason(1050):ssl/record/rec_layer_s3.c:1467:SSL 
alert number 50
ok 1 - ServerKeyExchange with 0 p
Segmentation Fault - core dumped
ok
All tests successful.

Regards,

Rainer