Question about SSL_key_update

[Benjamin Kaduk]

On Thu, Jul 09, 2020 at 06:07:41PM +0000, Andreas Müller wrote:
> Hi,
> 
> I "inherited" our project to support/use TLSv1.3 from a late colleague. We
> have a server written in C++ (Windows, Linux)
> and clients (Windows, Linux, also written in C++ and also a Java client).
> With Java, we use the native SSLSocket implementation, in Windows we use
> Schannel and in Linux we use OpenSSL 1.1.1g. It seems to work and even
> interoperability
> did not show issues on some initial testing.
> 
> I was curious about SSL_key_update. I read that other implementations use it
> automatically, but with OpenSSL I have to
> do it manually. So I added a
> 
>         int rc = SSL_key_update(ssl, SSL_KEY_UPDATE_REQUESTED);
> 
> after 1000 I/O operations and looked what happened.
> 
> I started with the Java client and it gets wrong application data in such a
> situation.
> 
> I tested with our Windows implementation (I know it may have
> interoperability issues) and here I can see the
> data I get from the server side and it is the same that I see on server side
> in the callback, but the Windows
> DecryptMessage function fails with SEC_E_INVALID_TOKEN. (I had expected
> something like SEC_I_RENEGOTIATE to
> get the information to put this record into InitializeSecurityContext.)
> 
> The Linux client also aborts the connection, but here I have not yet more
> details, but either the decryption fails
> or the decrypted application data is wrong. Hopefully I can dig in there
> next week.
> 
> Here is what happens on server side:
> 
> 1. I call SSL_key_update (see above)
> 2. I call SSL_write with application data
> 3. The write callback is invoked with this data:
>    DATA: 17 03 03 00 16 b6 02 b1 06 87 f2 30 0d 77 35 31 7a 5b 6d 3c c2 aa
> 11 2c 32 95 a9
> 4. The write callback is invoked again with application data provided to
> SSL_write:
>    DATA: 17 03 03 00 45 12 24 e5 66 36 2f 28 ea 62 2e 17 4c 62 f0 38 07 7f
> 72 70 87 25 ba 45 ff cf f7 9f 0d 7d 48 ...
> 
> I see these data arrive at the client side (Windows):
>    DATA: 17 03 03 00 16 b6 02 b1 06 87 f2 30 0d 77 35 31 7a 5b 6d 3c c2 aa
> 11 2c 32 95 a9
> but get the error described above. In Java I get wrong application data, so
> it seems to decrypt this as application
> data?!
> 
> I saw an additional call to SSL_do_handshake in the apps/s_server.c and
> tried this, but it did not change anything.
> 
> Is there anything else (except calling SSL_key_update) I have to take care of?

Per the documentation (https://www.openssl.org/docs/man1.1.1/man3/SSL_key_update.html)
it sounds like you're doing all you need to be.  (In particular, you don't need
to call SSL_do_handshake().)

Probably the fastest way to track down what's happening is to get a full packet
capture and keylog (e.g., with s_client -keylogfile path/to/log, then point wireshark
at the trace+keys.

-Ben