[openssl-users] Increasing the server write buffer for handshakes in 1.1.0

[Brandon Black]

Hi all,

  I'm running into an issue where if the server handshake response
exceeds some value a little over 4K (which is pretty easy these days
with a typical public cert, intermediate, and stapled OCSP response),
we're suffering an extra RTT in our SSL negotiations with
OpenSSL-1.1.0 (vs 1.0.2).  The server software is nginx.  You can see
our internal ticket with more detail at:
https://phabricator.wikimedia.org/T150561 .

This same issue was already raised and fixed with nginx years ago
against older OpenSSL versions in
https://trac.nginx.org/nginx/ticket/413 .

I suspect the workaround implemented at the time (which is obviously
not an ideal use of the APIs to begin with, with that wbio vs rbio
pointer comparison...) no longer works for 1.1.0.  I've tried
unconditionally calling BIO_set_write_buffer_size() in the same
callback as well, but it didn't improve the situation.

Is there an appropriate way to use the API to work around the write
buffer limit at handshake time for a server application with 1.1.0,
that we could patch up nginx with?

Another alternative would be to raise the default buffer size to 8K to
be more reflective of modern conditions.  I've made such a commit at
https://github.com/blblack/openssl/commit/5c3f1e46b61db591ea61d560ee51535286afa1a4
, but I haven't filed a pull request yet as I'm unsure on a couple of
fronts here:

(1) Whether there's an easier answer for server software developers
within the existing APIs (the main question in this post)

(2) With the default also currently being reused as the minimum
possible buffer size, I'm not sure whether it would be acceptable to
raise the minimum to 8K as well when changing the default. Splitting
the two would be a bit more invasive.


Thanks,
-- Brandon