[openssl-users] OpenSSL non-blocking epoll hanging on data receiving

[Matt Caswell]

On 27/05/16 15:51, counterpoint wrote:
> Seems to always be zero, Matt, as it should be. This gives some idea of
> what is going on:
> 
> Breakpoint 4, dcb_basic_read_SSL (dcb=0x7fffdc0158d0,
>     nsingleread=0x7fffffff86fc) at /root/MaxScale/server/core/dcb.c:1218
> 1218        switch (SSL_get_error(dcb->ssl, *nsingleread))
> $352 = 16384
> 
> Breakpoint 5, dcb_basic_read_SSL (dcb=0x7fffdc0158d0,
>     nsingleread=0x7fffffff86fc) at /root/MaxScale/server/core/dcb.c:1222
> 1222            MXS_DEBUG("%lu [%s] Read %d bytes from dcb %p in state %s "
> Fri May 27 16:47:12 BST 2016
> $353 = "Successful SSL read"
> 
> Breakpoint 1, dcb_read_SSL (dcb=0x7fffdc0158d0, head=0x7fffffff87d0)
>     at /root/MaxScale/server/core/dcb.c:1191
> 1191        ss_dassert(gwbuf_length(*head) == (start_length + nreadtotal));
> $354 = 0
> 
> Breakpoint 9, dcb_write (dcb=0x678ef0, queue=0x67d300)
>     at /root/MaxScale/server/core/dcb.c:1370
> 1370        below_water = (dcb->high_water && dcb->writeqlen < dcb->high_water);
> Fri May 27 16:47:12 BST 2016
> $355 = 16384
> $356 = "Writing to the client"
> 
> Breakpoint 4 shows the number of bytes read. Breakpoint 5 shows the case
> for analysis of the return code from the read. Breakpoint 1 shows the
> value of s->s3->rbuf.left Breakpoint 9 shows the number of bytes being
> passed to the write function.

So, if I understand you correctly, s->s3->rbuf.left is always 0
(indicating that there is no unprocessed data that OpenSSL has buffered)
and SSL_pending() returns 0 (indicating that OpenSSL has no processed
data bufferred). And this is the point where the hang in epoll occurs?

It sounds to me like OpenSSL on the server is behaving as expected. Its
processed all the data it has received and sent it on to your
application. This looks like either:
- the client hasn't sent the data that you're expecting to receive
(perhaps because of a problem on the client side)
or
- your server application hasn't correctly responded to something that
the client sent you.

Either way it looks to me like an application protocol level issue
rather than an SSL/TLS level issue. So the question is what data are you
waiting for from the client at this point? Does the client think it sent
it? Perhaps its sitting in some buffer somewhere. You need to work out
at the application protocol level what state things are in, and why the
client and server are apparently out of sync with each other.

Matt