[openssl-users] BIO_read hangs, how can I know if the server wants to send data?

[Michael Wojcik]

> From: Hanno Böck [mailto:hanno at hboeck.de]
> Sent: Tuesday, April 26, 2016 12:13
> To: Michael Wojcik
> Cc: openssl-users at openssl.org
> Subject: Re: [openssl-users] BIO_read hangs, how can I know if the server
> wants to send data?
> 
> Thanks for both your answers, that was very helpful (although it
> probably means what I'm trying to do is more complicated than I
> thought)...

My suggestion: Use an HTTP library such as libcurl. libcurl already supports integration with OpenSSL. Don't reinvent the HTTP wheel.

> One more question you might be able to answer:
> When I run my test code and connect to google.com I get the following
> bytes read for each BIO_read call:
> 1024
> 365
> 289

So Google is sending three TLS records with application data. That means it's doing the Right Thing: coalescing outbound data into a few messages. Without a wire trace, we can only guess what those three are.

> When I run these against my own server (relatively standard
> apache2.4+openssl setup) I get very different numbers:
> 240
> 287
> 2
> 588
> 2
> 41
> 2

Ugh. Apache is doing the Wrong Thing. It's sending data as it generates it, instead of coalescing. Those two-octet messages are almost certainly the CR LF pairs that terminate lines of the HTTP header.

Why is this wrong? Because sending short TCP packets is inefficient, and can trigger Nagle / Delayed ACK Interaction, which rate-limits the traffic. An application can prevent NDAI by disabling Nagle, but in most cases that's just a sign that the application developer doesn't know how to use TCP properly.

The problem is reduced a bit by using TLS, because the TLS record and encryption overhead make some smaller packets bigger than the MSS, and thus not affected by Nagle. But it's still bad. (And from the application developer's point of view, TLS libraries typically make the problem harder to resolve, due to a lack of a gather-send operation.)
 
> Why is this so much more split up? And to what correspond these
> BIO_read chunks on the protocol level? Are these TLS records? TCP
> packets? Is there something horribly wrong with my server config
> because it splits them up in so many small parts?

I don't know enough about Apache configuration to say. (If Apache routinely does this, though, it's no wonder there are all those benchmarks showing better performance with Nginx. This is Sockets 101 stuff.)

In any case it's not "horribly wrong". It's just costing you bandwidth and latency.

-- 
Michael Wojcik
Technology Specialist, Micro Focus