[openssl-users] Strange problem with 1.0.2f SSL_shutdown in multithreaded server

[Jakob Bohm]

I am trying to upgrade an existing 3rd party multithreaded server
from OpenSSL 1.0.2c to 1.0.2f .  However when I do so, it starts
mishandling the close_notify "alert".

1.0.2f seems to send the close_notify alert unencrypted followed
by an encrypted decrypt_failed alert, where 1.0.2c correctly
sends just an encrypted close_notify alert.

I am unsure if this exposed a bug in the daemon or in OpenSSL
itself.

P.S. The "block cipher pad is wrong:s3_pkt.c:452" error message
really means "unencrypted parts of encrypted record are invalid
for encryption algorithm" (ssl3_enc->enc() returned 0 because an
AES-CBC encrypted block cannot be be 2 bytes long).

Notes on the daemon:

- It is the same binary in both tests, compiled against 1.0.2a,
  but loading either the OpenSSL 1.0.2c or 1.0.2f shared object.

- It is multithreaded and does set up a thread id callback and
  a locking callback (but not a dyn locking callback).  The
  thread id callback uses the 0.9.8 compatible thread callback
  API, and I have checked that the thread_id is actually an
  unsigned long value.

- Tests below were done with TLS 1.0 to keep the log shorter,
  but the same error occurs with TLS 1.2.

- The error does not occur against 1.0.2 s_server, which I
  suppose is single_threaded.

- OS is Linux x86_64.

Output from 1.0.2f s_client against 1.0.2f daemon:

read from 0xbd4780 [0xbda813] (5 bytes => 5 (0x5))
0000 - 15 03 01 00 02 .....
<<< ??? [length 0005]
     15 03 01 00 02
read from 0xbd4780 [0xbda818] (2 bytes => 2 (0x2))
0000 - 01                                                .
0002 - <SPACES/NULS>
 >>> ??? [length 0005]
     15 03 01 00 20
write to 0xbd4780 [0xbded63] (37 bytes => 37 (0x25))
0000 - 15 03 01 00 20 dd ce 72-24 f3 d6 2f e1 12 72 e3 .... ..r$../..r.
0010 - 89 53 f0 ef 28 da 7d d3-76 c3 f4 2c a8 af 12 2f .S..(.}.v..,.../
0020 - 0a 01 2e e0 a5 .....
 >>> TLS 1.0 Alert [length 0002], fatal decryption_failed
     02 15
140649912034984:error:1408F081:SSL routines:SSL3_GET_RECORD:block cipher 
pad is wrong:s3_pkt.c:452:
 >>> ??? [length 0005]
     15 03 01 00 20
write to 0xbd4780 [0xbded63] (37 bytes => -1 (0xFFFFFFFFFFFFFFFF))


Output from 1.0.2f s_client against 1.0.2c daemon:

read from 0x10e6780 [0x10ec813] (5 bytes => 0 (0x0))
read:errno=0
 >>> ??? [length 0005]
     15 03 01 00 20
write to 0x10e6780 [0x10f0d63] (37 bytes => 37 (0x25))
0000 - 15 03 01 00 20 db 1d f3-2e 24 a6 ae 93 27 05 67 .... ....$...'.g
0010 - b2 0e 61 5f 11 32 83 32-3e 55 d9 e9 0b c2 39 34 ..a_.2.2>U....94
0020 - f0 46 70 8f 16 .Fp..
 >>> TLS 1.0 Alert [length 0002], warning close_notify
     01 00



Enjoy

Jakob
-- 
Jakob Bohm, CIO, Partner, WiseMo A/S.  https://www.wisemo.com
Transformervej 29, 2860 Søborg, Denmark.  Direct +45 31 13 16 10
This public discussion message is non-binding and may contain errors.
WiseMo - Remote Service Management for PCs, Phones and Embedded