Replay HTTP traffic

[Matt Caswell]

On 17/05/2021 14:41, Григорий Сморкалов wrote:
> Hello. I am trying to debug some ssl related code and I need some help.
> 
> We have a HTTP client based on libuv and libopenssl for TLS. It is an 
> internal C++ library with its own TCP wrapper around lubuv and HTTP 
> parser. It works fine and our servers make millions of HTTPS requests to 
> social networks with it. If it is one connection per request 
> (Connection: Close) there is no problem at all. But sometimes 
> connections with keep-alive receive strange ssl errors:
> 
> error:04067084:rsa routines:rsa_ossl_public_decrypt:data too large for 
> modulus , or error:04067072:rsa routines:rsa_ossl_public_decrypt:padding 
> check failed
> 
> It is a really rare event, once per million I think. The error is 
> returned from SSL_read when new data comes from the server. It is never 
> the first response, usually there are more than ten requests/responses 
> in the connection before the error.

If I understand you correctly then you are seeing this at some point 
*after* the initial handshake and the connection has been running for a 
while.

If so that is a very strange error indeed. These are RSA errors. But 
once the initial handshake is complete there should be no reason for 
libssl to be performing RSA calls. RSA is *only* used during the 
handshake and not during application data transfer. Unless that is there 
is a reneg handshake happening (only applies for TLSv1.2 or less)...but 
if so that would be fairly clear in the wireshark logs.

Does your application do anything with libcrypto directly? Or does it 
only use libssl?

Is your client application multi-threaded?

I'm wondering whether this error is actually a stale error left in the 
queue from some earlier problem on the same thread.

You could try forcefully clearing any stale errors (ERR_clear_error()) 
before any SSL_read() calls and see if the problem goes away.


> 
> We have a tcpdump of such connections and keylog made with 
> SSL_CTX_set_keylog_callback. Wireshark opens this dump and decrypts it 
> normally using keylog as pre-master keyfile. The last packet produces an 
> error in our HTTP client but in wireshark it is ok and it contains 
> normal HTTP response with 200 OK. No sign of any error or data 
> corruption. That fact makes me think that data is ok and my openssl 
> usage has some problems.
> 
> I want to reproduce this situation and replay this tcpdump. It means run 
> our server (actually only http client part) and give it captured data. 
> It is no problem to make a server that sends exactly the same data from 
> tcp dump. It is no problem to make exactly the same http request. But I 
> need to use the pre-master key from the keylog on the client side. I 
> cannot find any function that sets keys to SSL_CTX* or SSL*. Is there any?
> 
> I tried to build my own libopenssl with constant keys. I put 
> memcpy(s->session->master_key, overriden_secret, 48) in 
> ssl_generate_master_secret and tls13_generate_secret. Also 
> memcpy(s->s3->client_random, overriden_random, 32); in 
> tls_construct_client_hello and tls_early_post_process_client_hello. It 
> doesn't work and produces ssl error on handshake phase 
> error:1416C095:SSL routines:tls_process_finished:digest check failed. 
> Client Hello produced by this patched libopenssl is always different, 
> this means I haven't replaced all keys. It is something in s->tmp 
> structure I cannot understand to replace all usages and values.

This is very much a non-trivial task. OpenSSL has no support for this 
kind of thing at the moment and it would be difficult to add it. I don't 
think the key logging logs the ephemeral keys that are used as input to 
the master secret generation. So the ClientHello key_share (assuming 
TLSv1.3) is going to be different regardless.

You could conceivably hack the finished check so that it passes 
regardless if you still end up with the right master secret - but 
everything isn't necessarily going to be the *same* as in the initial 
failing run. If my possibly theory about a stale error on the queue is 
right then even if you got everything right then it might still not show 
up the problem if the stale error is related to some other thing that 
happened on the same thread.

Matt



> 
> Is there a simpler way?
> 
> Without reproducing it is practically impossible to find a bug. Even if 
> it does not reproduce, I'll get some information. Maybe it is UB in a 
> different place.
> 
> I've asked the same question on stackoverflow, so you can answer there 
> if it is easier or better for you: 
> https://stackoverflow.com/questions/67570255/how-to-replay-encrypted-traffic-with-libopenssl 
> <https://stackoverflow.com/questions/67570255/how-to-replay-encrypted-traffic-with-libopenssl>
> 
> Thank you!
> 
>