[openssl-users] SSL_read() returns -1, and SSL_read_ex does not update readbytes where a record containing a session ticket is being read (TLS 1.3)

[Arran Cudbard-Bell]

> On 23/01/2019 14:04, Arran Cudbard-Bell wrote:
>> I'm working with wpa_supplicant to try and fix up its EAP-TTLS and EAP-PEAP
>> implementations to work correctly with TLS 1.3 and session tickets.
>> 
>> Where a new_session_ticket message is sent after client/server finish, calls
>> to SSL_read() result in the new_session_ticket message being processed
>> correctly, but SSL_read() returns -1 if no application_data is available in
>> the input BIO. SSL_read_ex() returns 0, but readbytes isn't updated to
>> reflect the number of bytes consumed whilst processing the session tickets.
> 
> This is intended behaviour. SSL_read() returns the number of plaintext
> application data bytes that have been populated in the provided buffer
> (similarly for readbytes).

OK, confirmed.  There was something in our code base that lead me to believe SSL_read() was returning the number of bytes consumed when processing handshake messages. I've just confirmed this is not the case and it always returns -1 during the handshake phase.

> If no application data is available then you get the
> -1 (for SSL_read()) or 0 (for SSL_read_ex()) return code. You also get that
> return code for other types of issues, and you are supposed to call
> SSL_get_error() to interpret it.

In this case SSL_get_error() returns 2 (SSL_ERROR_WANT_READ) with either SSL_MODE_AUTO_RETRY on or /off.

> Note that by default OpenSSL sets SSL_MODE_AUTO_RETRY in 1.1.1 (which it didn't
> in 1.1.0). This should mean that it automatically tries again to read
> application data without returning an error. Have you switched
> SSL_MODE_AUTO_RETRY off?

It was left at defaults for the initial tests.

>> It seems to be that SSL_read() should return a positive integer representing
>> the number of bytes read from the BIO whilst processing the session tickets,
>> and SSL_read_ex should update readbytes to the number of bytes read from the
>> BIO whilst processing the session tickets, as is done with other handshake
>> messages.
> 
> This is not correct. SSL_read()/SSL_read_ex() only ever provide the number of
> application data bytes that have been read, irrespective of how many bytes were
> read from the underlying BIO.

Would call BIO_pending() before/after the call to SSL_read() be the best way to determine the number of bytes consumed by SSL_read()?

We could use this to determine what SSL_ERROR_WANT_READ is indicating.  As it seems  SSL_ERROR_WANT_READ could indicate two conditions in this scenario:

1) No pending bytes - Additional handshake messages were processed, there's an expectation of additional application_data, but no hint that more application_data will be forthcoming.
2) Pending bytes - There is an incomplete record that needs processing.  Additional data should be fed into the BIO.

-Arran