[openssl-dev] why TLSv1 need two tls1_enc to get decrypted data while TLSv1.1/TLSv1.2 need one in OpenSSL1.1.0f?

[Matt Caswell]

On 27/09/17 15:44, Ma chunhui wrote:
> Hi, 
> 
> I met one problem when using OpenSSL1.1.0f with protocol TLSv1.
> In brief, when using TLSv1,  after server side received encrypted data,
> and after function tls1_enc finished, the decrypted data is not put in
> result buffer, after another tls1_enc, the decrypted data is put in
> result buffer. While TLSv1.1/TLSv1.2 needs only one tls1_enc.
> 
> 
> The way to reproduce it is quite simple:
> 
> 1.some preparation: openssl req -x509 -newkey rsa:2048 -keyout key.pem
> -out cert.pem -days 365 -nodes
> 2.start server: openssl s_server -key key.pem -cert cert.pem -accept
> 44330 -www
>     it's better to start server with gdb, and set breakpoints at
> tls1_enc, then continue to run. 
> 3.openssl s_client -connect localhost:44330 -tls1 -debug
> 
> After the client is started,  the server side will stop at breakpoint,
> do several "c" to make it continue to run to wait for client's messages
> Then at client side, type a simple "hello" message and press Enter. Then
> server side will stop at tls1_enc, the input data is same as encrypted
> data from client side, but after Evp_Cipher and some pad removing, the
> decrypted data length is 0. After another tls1_enc, the decrypted data
> "hello" is put in the result buffer.
> 
> But if client use -tls11 or -tls12, the decrypted "hello" is put in the
> result buffer after the first tls1_enc.
> 
> Could anyone explains why the behavior of decryption is different
> between TLSv1 and TLSv1.1/TLSv1.2?

In TLSv1 and below the CBC IV is the previous record's last ciphertext
block. This can enable certain types of attack where an attacker knows
the IV that will be used for a record in advance. The problem was fixed
in the specification of TLSv1.1 and above where a new IV is used for
each record. As a counter measure to this issue OpenSSL (in TLSv1) sends
an empty record before each "real" application data record to
effectively randomise the IV and make it unpredictable so that an
attacker cannot know it in advance.

Therefore a TLSv1 OpenSSL client will send two records of application
data where a TLSv1.1 or above OpenSSL client will just send one. This
results in tls1_enc being called twice on the server side.

This behaviour can be switched off by using the option
SSL_OP_DONT_INSERT_EMPTY_FRAGMENTS - but since this is considered
insecure that would probably be unwise.

Matt