[openssl-users] SSL_shutdown return error when close in init

[Linsell, StevenX]

On Tue, 18 Apr 2017, mid_li at 163.com wrote:
>Hello
> I'm using open1.1.0e in async mode with intel QuickAssist Engine to handle https connections? but there's some problem.
>
>client(ab)-------------------------- server(my program)
>
><---------TCP handshake----------------> -------------ssl client hello---------------> <---------server hello,certicate...--------- -----------client key exchange....--------> >//here, server's SSL_do_handshake reutrns SSL_ERROR_WANT_ASYNC repeatly,
>
>-----------FIN+ACK---------------------->
>
>//client want to close the connection, then, server should close ssl connection ,In program, I intend to close SSL connections in quiet mode?
>SSL_set_quiet_shutdown(ssl,1);
>SSL_shutdown(ssl);
>
>but SSL_shutdown returns SSL_ERROR_SSL, because SSL_in_init(s) return true.
>
>I'm confused, what should I do here ???
>(1) just call SSL_free(ssl) to free SSL connection, then the async engine may callback and using SSL's waitctx, which cause crash.  Also I noticed that SSL's job >doesn't free neither, which may cause memory leak;
>
>(2)continue call SSL_shutdown(ssl),  and it will always return SSL_ERROR_SSL
>
>Is anybody know? thanks  

The root cause of the issue is that it is not valid to move state from init to shutdown when there is still an asynchronous operation in progress.
The fact that the client wants to close the connection should be saved, the asynchronous operation should be completed (keep calling SSL_do_handshake until SSL_get_error does not return SSL_ERROR_WANT_ASYNC) then based on what you saved do the same behaviour you would have done in the case of the client wanting to close the connection if you are running synchronously.
As long as you have completed the asynchronous operation then there will be no problem calling SSL_free on the connection as there will be no callback that will run later.
By continuing to call SSL_do_handshake until the sync job completes all you are doing is running the SSL_do_handshake to the same point it would have returned and detected the error if you were running synchronously.
Note that it is never valid to call SSL_do_handshake(), start an asynchronous operation (SSL_get_error returning SSL_ERROR_WANT_ASYNC), then transition straight to calling a different asynchronous enabled function like SSL_shutdown(). If you do that you will find that when you call SSL_shutdown it will detect there is already an async job in progress and will context switch into that job rather than starting an async job for the SSL_shutdown behaviour. In other words you will end up running SSL_do_handshake code when you think you are running SSL_shutdown code. Even worse they may have completely different return behaviour so you get an unexpected result. The OpenSSL documentation makes it clear that you must keep calling the same asynchronous function with the same parameters until the async job has completed.

Hope that helps,

Steve Linsell                                     Intel Shannon DCG/CID Software Development Team
Stevenx.Linsell at intel.com

--------------------------------------------------------------
Intel Research and Development Ireland Limited
Registered in Ireland
Registered Office: Collinstown Industrial Park, Leixlip, County Kildare
Registered Number: 308263


This e-mail and any attachments may contain confidential material for the sole
use of the intended recipient(s). Any review or distribution by others is
strictly prohibited. If you are not the intended recipient, please contact the
sender and delete all copies.