Determine that there is no forward progress with non blocking SSL socket

[Michael Wojcik]

From: openssl-users [mailto:openssl-users-bounces at openssl.org] On Behalf Of Eran Borovik
Sent: Monday, January 27, 2020 07:07

> When do I stop? what is the best way to actually determine there can
> be no more forward progress both on the send and the receive side, and
> epoll must be used?

...

> I am afraid that if the send buffer is full and there is nothing to receive,
> I will keep getting WANT_READ for receive, and WANT_WRITE for send until
> actual data arrives or can be sent which defeats the purpose of epoll.

The following is untested and off the top of my head.

Think of it this way. You have these possible states:

1. You don't want to do anything with the conversation. It can be closed.
2. You want to receive. There are three possible causes:
  - You'd like to receive data (or close/error indication) from the peer.
  - SSL_send returned WANT_READ.
  - Both of the above are true.
3. You want to send. There are three possible causes:
  - You have buffered outbound application data (or shutdown/close) which you
    haven't been able to send yet.
  - SSL_receive returned WANT_WRITE.
  - Both of the above are true.
4. You want to receive and to send.

So, keep track of the above using a per-conversation pair of want_to_send and want_to_receive variables and use the following algorithm:


   /* any time we buffer outbound data, set want_to_send = 1 */
   want_to_receive = 1 /* at start of conversation, we want data from peer */

   while want_to_send or want_to_receive
      {
      /* Wait until we can do something we want to do */
      do_recv = false, do_send = false
      if want_to_send and want_to_receive
         {
         epoll until socket is readable or writable
         do_recv = readable
         do_send = writable
         }
      else if want_to_send
         {
         epoll until socket is writable
         do_send = true
         }
      else if want_to_receive
         {
         epoll until socket is readable
         do_recv = true
         }

      /* Now perform I/O */
      if do_recv
         {
         SSL_read(...)
         if WANT_WRITE
            want_to_send = true
         else if WANT_READ
            want_to_receive = true /* couldn't send enough to clear WANT_WRITE */
         else if error
            ...
         else
            want_to_receive = false /* maybe, depending on application */

         process_received_data(...)
         }
      if do_send
         {
         SSL_write(...)
         if WANT_READ
            want_to_receive = true
         else if WANT_WRITE
            want_to_send = true /* couldn't send enough to clear WANT_WRITE */
         else if error
            ...
         else
            {
            update send buffer info
            if all buffered data sent
               want_to_send = false /* any pending WANT_WRITE should now be clear */
            }
         }
      }
   close conversation

Now, in practice, you probably always want to poll on readability even if you're not
expecting data from the peer, because you'll want to be notified of peer close. And
you may want to be able to break out of an epoll for readability only in order to send
data, depending on your application design and requirements. You can do that using a
timeout and polling the want_to_send state variable, or using an internal channel such
as a pipe or socket which the polling thread adds to the epoll readable-descriptor set,
and the sending thread writes a wakeup message to.

It's an open design question if you want to set want_to_receive to false in the do_recv
body, as I have it above, and then set it back to true if you decide you're not done
with the conversation in process_received_data; or leave it set to true and set it to
false only when you've received a close indication from the peer; or possibly some
other behavior that's appropriate for your application.


In brief: at the top of the loop, figure out if you want to try to send, receive, or
both. The types of I/O you want to do is the union of the application state and the
OpenSSL WANT_* state. Then poll for all the types of I/O you want, and when the socket
is available for any of them, attempt that type. Deal with any successful operation,
and loop.


In the "problem case" you described above, this code will discover that you want to
receive and to send, so it will first poll for either type of availability on the
socket. If the socket ends up readable, you'll do a receive, which may succeed or
may return WANT_WRITE or may indicate a peer close or may indicate an error. It could
also return WANT_READ if it's not able to receive enough TLS data to satisfy a pending
WANT_READ. In that case we keep want_to_receive set and go around the loop again.

And if the socket is writable (regardless of whether it's also readable), it will try
to send some of your application data; that will also try to handle any pending
WANT_WRITE condition. The result might be sending all the application data, sending
some or none of the application data, satisfying a WANT_WRITE condition, failing to
send enough to satisfy the WANT_WRITE, getting an error, etc. If we do send some
application data, we can assume any pending WANT_WRITE from the previous receive
operation was satisfied.

--
Michael Wojcik
Distinguished Engineer, Micro Focus