[openssl-users] Strange problem with 1.0.2f SSL_shutdown in multithreaded server

[Matt Caswell]

On 02/02/16 07:52, Jakob Bohm wrote:
> I am trying to upgrade an existing 3rd party multithreaded server
> from OpenSSL 1.0.2c to 1.0.2f .  However when I do so, it starts
> mishandling the close_notify "alert".
> 
> 1.0.2f seems to send the close_notify alert unencrypted followed
> by an encrypted decrypt_failed alert, where 1.0.2c correctly
> sends just an encrypted close_notify alert.
> 
> I am unsure if this exposed a bug in the daemon or in OpenSSL
> itself.

I have a theory.

Previous versions of 1.0.2 handled an SSL_shutdown() call while in the
middle of a handshake by ignoring it and returning 1 immediately
(meaning everything shutdown successfully). Clearly everything did not
shutdown successfully so the return value is not correct.

1.0.2f "fixed" this to attempt to more appropriately shutdown the
connection. It now attempts to send a close_notify alert immediately
(and will return 0 - meaning more work to do), it will then fail with a
-1 return on a second call to SSL_shutdown (meaning we failed to
complete a bi-directional shutdown). The reason is that completing a
full bi-directional shutdown while mid-handshake is problematic (e.g. if
the peer has sent a CCS before it receives our close_notify).

My theory as to what is happening in this case is that SSL_shutdown is
being called mid-handshake. Could that be the case do you think? It
doesn't entirely match with the description you gave (i.e. 1.0.2c would
not send an encrypted close_notify...it wouldn't send anything in this
scenario)...but its the only change I can think of that might have a
bearing on this.

If so then I could see a situation where the server has not yet
processed a CCS from the client and SSL_shutdown is called. It therefore
sends its close_notify in the clear. In the meantime the client has sent
its CCS before processing the close_notify. The client then expects the
next message from the server to be encrypted...which it obviously isn't.

It's not entirely clear to me what the correct behaviour is in that
situation, but it is certainly unexpected and a change from what
happened before.

It looks to me that trying to send a close_notify mid-handshake was
probably a bad idea. Perhaps a better fix is just to have SSL_shutdown()
return -1 and do nothing if called mid-handshake.

I've attached a 1.0.2 patch that does this. Would you be able to try it,
and see if that corrects your problem?

Thanks

Matt
-------------- next part --------------
A non-text attachment was scrubbed...
Name: shutdown-1.0.2.patch
Type: text/x-patch
Size: 3888 bytes
Desc: not available
URL: <http://mta.openssl.org/pipermail/openssl-users/attachments/20160202/89ed97fd/attachment-0001.bin>