Question about SSL_key_update

[Andreas Müller]

Hi,

I "inherited" our project to support/use TLSv1.3 from a late  
colleague. We have a server written in C++ (Windows, Linux)
and clients (Windows, Linux, also written in C++ and also a Java  
client). With Java, we use the native SSLSocket implementation, in  
Windows we use Schannel and in Linux we use OpenSSL 1.1.1g. It seems  
to work and even interoperability
did not show issues on some initial testing.

I was curious about SSL_key_update. I read that other implementations  
use it automatically, but with OpenSSL I have to
do it manually. So I added a

         int rc = SSL_key_update(ssl, SSL_KEY_UPDATE_REQUESTED);

after 1000 I/O operations and looked what happened.

I started with the Java client and it gets wrong application data in  
such a situation.

I tested with our Windows implementation (I know it may have  
interoperability issues) and here I can see the
data I get from the server side and it is the same that I see on  
server side in the callback, but the Windows
DecryptMessage function fails with SEC_E_INVALID_TOKEN. (I had  
expected something like SEC_I_RENEGOTIATE to
get the information to put this record into InitializeSecurityContext.)

The Linux client also aborts the connection, but here I have not yet  
more details, but either the decryption fails
or the decrypted application data is wrong. Hopefully I can dig in  
there next week.

Here is what happens on server side:

1. I call SSL_key_update (see above)
2. I call SSL_write with application data
3. The write callback is invoked with this data:
    DATA: 17 03 03 00 16 b6 02 b1 06 87 f2 30 0d 77 35 31 7a 5b 6d 3c  
c2 aa 11 2c 32 95 a9
4. The write callback is invoked again with application data provided  
to SSL_write:
    DATA: 17 03 03 00 45 12 24 e5 66 36 2f 28 ea 62 2e 17 4c 62 f0 38  
07 7f 72 70 87 25 ba 45 ff cf f7 9f 0d 7d 48 ...

I see these data arrive at the client side (Windows):
    DATA: 17 03 03 00 16 b6 02 b1 06 87 f2 30 0d 77 35 31 7a 5b 6d 3c  
c2 aa 11 2c 32 95 a9
but get the error described above. In Java I get wrong application  
data, so it seems to decrypt this as application
data?!

I saw an additional call to SSL_do_handshake in the apps/s_server.c  
and tried this, but it did not change anything.

Is there anything else (except calling SSL_key_update) I have to take care of?

Did anyone use this successfully with a Java SSLSocket? (We currently  
use Azul 8 which has a backport of TLSv1.3, and Azul 11).

Has anyone experience with Windows Schannel against OpenSSL?

I still believe that I do something wrong, because the Linux client  
also fails.

Any hints are appreciated.


Regards,
Andreas