[openssl-dev] [openssl.org #3712] TLS Renegotiation with Java is broken

Wed Feb 18 20:12:09 UTC 2015

I ran into this problem while connecting to a PostgreSQL server (PostgreSQL uses OpenSSL
for SSL support) with a Java client using the PostgreSQL JDBC driver (which uses
the Java Secure Socket Extension which is part of Oracle's Java Runtime Environment).

Since database connections are potentially long-lived, the PostgreSQL server will
trigger a renegotiation after a certain amount of data has been exchanged via the
TLS channel; this amount is configurable with the parameter "ssl_renegotiation_limit".

This renegotiation is always aborted by OpenSSL with the error "unexpected record".
I could reproduce the problem with OpenSSL 1.0.1e on Linux and OpenSSL 1.0.1j on
Windows using Oracle JRE 1.7.0_71 and 1.7.0_75 on the client side.
The protocol version in effect is TLS 1.2 (0x303).

I will describe the problem in more detail below, but my conclusion is that OpenSSL
violates RFC 5246 (TLS 1.2) and RFC 4346 (TLS 1.1) since it chokes on an "Application
Data" record received right after it has sent a "Server Hello Done" record.
At this point OpenSSL waits for a "Client Certificate" or a "Client Key Exchange" record
and is not ready to receive application data.

However, RFC 5246 and RFC 4346 state at the end of chapter 6.2.1:

   Note: Data of different TLS Record layer content types MAY be
   interleaved.  Application data is generally of lower precedence for
   transmission than other content types.  However, records MUST be
   delivered to the network in the same order as they are protected by
   the record layer.  Recipients MUST receive and process interleaved
   application layer traffic during handshakes subsequent to the first
   one on a connection.

The last sentence clearly puts the blame on OpenSSL.
This seems to be an addition in TLS 1.1 since it cannot be found in RFC 2246.

The problem is likely not restricted to PostgreSQL, but a generic problem that
will affect all renegotiations between JSSE and OpenSSL.
The only workaround is to disable renegotiation, but that will reduce the
security of the communication.

Yours,
Laurenz Albe

Detailed analysis using OpenSSL 1.0.1e on RedHat Enterprise Linux 6:

The OpenSSL error is thrown in this part of the code:
https://github.com/openssl/openssl/blob/master/ssl/s3_pkt.c#L1601
(the code does not seem to have changed substantially since 1.0.1e).

The stack trace at the time is like this:
#0  ssl3_read_bytes (s=0x228d6a0, type=22, buf=0x231ee20 "\016", len=4, peek=0) at s3_pkt.c:1421
#1  0x00000039d402dac2 in ssl3_get_message (s=0x228d6a0, st1=8576, stn=8577, mt=-1, max=102400, ok=0x7fffe6bcfe1c) at s3_both.c:457
#2  0x00000039d401e36a in ssl3_check_client_hello (s=0x228d6a0) at s3_srvr.c:873
#3  0x00000039d40219db in ssl3_accept (s=0x228d6a0) at s3_srvr.c:572
#4  0x00000039d402c9a8 in ssl3_read_bytes (s=0x228d6a0, type=23, buf=0xd9b4e0 "P", len=8192, peek=0) at s3_pkt.c:1342
#5  0x00000039d4027ec0 in ssl3_read_internal (s=0x228d6a0, buf=0xd9b4e0, len=8192, peek=0) at s3_lib.c:4273
#6  0x0000000000698df1 in be_tls_read (port=0x228d490, ptr=0xd9b4e0, len=8192) at be-secure-openssl.c:572
#7  0x00000000006880d3 in secure_read (port=0x228d490, ptr=0xd9b4e0, len=8192) at be-secure.c:135
[...]

s->version is 0x301 (TLS 1.2), and s->state is 0x2180 (SSL3_ST_SR_CERT_A).

I put SSL_CTX_set_info_callback and SSL_CTX_set_msg_callback into the PostgreSQL source,
and the output looks as follows:

info_callback: SSL_accept:SSL renegotiate ciphers
msg_callback: write 0x303 handshake 00...
info_callback: SSL_accept:SSLv3 write hello request A
info_callback: SSL_accept:SSLv3 flush data
info_callback: SSL_accept:SSLv3 write hello request C
info_callback: SSL_accept:before accept initialization
msg_callback: read 0x303 handshake 01...
info_callback: SSL_accept:SSLv3 read client hello A
msg_callback: write 0x303 handshake 02...
info_callback: SSL_accept:SSLv3 write server hello A
msg_callback: write 0x303 handshake 0B...
info_callback: SSL_accept:SSLv3 write certificate A
msg_callback: write 0x303 handshake 0C...
info_callback: SSL_accept:SSLv3 write key exchange A
msg_callback: write 0x303 handshake 0E...
info_callback: SSL_accept:SSLv3 write server done A
info_callback: SSL_accept:SSLv3 flush data
msg_callback: write 0x303 alert 02...
info_callback: SSL3 alert write:fatal:unexpected_message
info_callback: SSL_accept:error in SSLv3 read client certificate A

I used Wireshark to capture the network communication, and the result
looks like this (the offending "Application Data" record must be number 44):

No.  Source           Destination      Protocol Length Info
28   Java client      OpenSSL server   TLSv1.2  203    Application Data
29   OpenSSL server   Java client      TLSv1.2  235    Application Data
30   OpenSSL server   Java client      TLSv1.2  219    Application Data
31   Java client      OpenSSL server   TCP      54     49860→5432 [ACK] Seq=1199 Ack=3272 Win=65536 Len=0
32   OpenSSL server   Java client      TCP      1514   [TCP segment of a reassembled PDU]
33   OpenSSL server   Java client      TCP      1514   [TCP segment of a reassembled PDU]
34   Java client      OpenSSL server   TCP      54     49860→5432 [ACK] Seq=1199 Ack=6192 Win=65536 Len=0
35   OpenSSL server   Java client      TCP      1514   [TCP segment of a reassembled PDU]
36   OpenSSL server   Java client      TCP      1514   [TCP segment of a reassembled PDU]
37   OpenSSL server   Java client      TCP      1514   [TCP segment of a reassembled PDU]
38   OpenSSL server   Java client      TLSv1.2  1031   Application Data
39   Java client      OpenSSL server   TCP      54     49860→5432 [ACK] Seq=1199 Ack=11549 Win=65536 Len=0
40   OpenSSL server   Java client      TLSv1.2  139    Encrypted Handshake Message
41   OpenSSL server   Java client      TLSv1.2  1003   Application Data
42   Java client      OpenSSL server   TCP      54     49860→5432 [ACK] Seq=1199 Ack=12583 Win=64512 Len=0
43   Java client      OpenSSL server   TLSv1.2  443    Encrypted Handshake Message
44   Java client      OpenSSL server   TLSv1.2  203    Application Data
45   OpenSSL server   Java client      TCP      60     5432→49860 [ACK] Seq=12583 Ack=1737 Win=22144 Len=0
46   OpenSSL server   Java client      TLSv1.2  1514   Encrypted Handshake Message, Encrypted Handshake Message
47   OpenSSL server   Java client      TLSv1.2  342    Encrypted Handshake Message
48   OpenSSL server   Java client      TLSv1.2  139    Encrypted Alert
49   Java client      OpenSSL server   TCP      54     49860→5432 [ACK] Seq=1737 Ack=14416 Win=65536 Len=0
50   OpenSSL server   Java client      TCP      60     5432→49860 [FIN, ACK] Seq=14416 Ack=1737 Win=22144 Len=0
51   Java client      OpenSSL server   TCP      54     49860→5432 [ACK] Seq=1737 Ack=14417 Win=65536 Len=0
52   Java client      OpenSSL server   TLSv1.2  203    Encrypted Handshake Message
53   OpenSSL server   Java client      TCP      60     5432→49860 [RST] Seq=14417 Win=0 Len=0

This is the Java program I use to reproduce the problem:

public class PGConn {
    public static void main(String[] args) throws ClassNotFoundException, java.sql.SQLException, java.io.IOException {
        Class.forName("org.postgresql.Driver");
        java.sql.Connection conn = java.sql.DriverManager.getConnection(
            "jdbc:postgresql://hostname/dbname?user=username&password=pwd&ssl&sslfactory=org.postgresql.ssl.NonValidatingFactory");
        java.sql.Statement stmt = conn.createStatement();
        stmt.execute("SET ssl_renegotiation_limit='3kB'");
        stmt.executeQuery("SELECT repeat('0123456789', 900)").close();
        stmt.executeQuery("SELECT repeat('0123456789', 900)").close();
        conn.close();
    }
}