[openssl-dev] a possible bug for certain usage of openSSL1.1.0 with TLSv1

[Ma chunhui]

Hi, Openssl team
After we upgrade openSSL from 1.0.2 to 1.1.0f, we met an error which might
be a bug.

1. our usage
We're using java application, and using JNI call to call OpenSSL API.(In
fact, we're using an opensource project named wildfly-openssl)

After context and ssl is created, for one record, the decryption step is
like this:
call BIO_write to put encrypted data into OpenSSL
call SSL_read(ssl, buf=0x0, len=0) to read 0 byte to an empty address(Note,
this is a 0-length byte read)
use SSL_pending to check if there is any decrypted data in the result buffer
if there is data, use SSL_read to read pending data into dest buffer.

2. The usage works fine for openSSL 1.0.1 and 1.0.2 with all protocols. But
not work for OpenSSL 1.1.0 with TLSv1.

The reason is: TLSv1 needs two records for application data,  and
decrypting the first record will get a 0 length result buffer, and
decrypting the second record will get the final decrypted data in the
result buffer.

 In OpenSSL 1.0.1 & 1.0.2,  the first 0 length result buffer doesn't matter
but  but in OpenSSL 1.1.0  the first 0 length result buffer will stop the
next decryptions.

The process in OpenSSL1.1.0 is like this:
For the first record: after BIO_write , SSL_read(ssl, buf=0x0, length=0) is
called, the first record is decrypted, tls1_enc is executed and get a 0
length result buffer, and then in ssl3_get_record, numrpipes is set to 1.
Then after the execution return to ssl3_read_bytes, it found the user want
to read 0 length, so it directly returns. leaving the state of the result
buffer to un-read(don't have a chance to set the buffer to have been read).
Then for the second record, before ssl3_get_record, it first check
numrpipes, and it found the numrpipes is not 0, and last buffer is not
read, so it won't process the second record.  and This cause fails.

and in OpenSSL 1.0.2, the condition of ssl3_get_record is if (rr->length ==
0 || xxx), so ssl3_get_record can process the second record.

Of course, I can change our usage to let it work for OpenSSL 1.1.0, for
example, we can call SSL_read even we found SSL_pending is 0. But that
means we may need another JNI call, which may affect our performance (we
don't want to remove the 0 length SSL_read)

A possible fix might like this:
+++b/ssl/record/rec_layer_s3.c
@@ -1132,6 +1132,12@@int ssl3_read_bytes(SSL *s, int type, int *recvd_type,
unsigned char *buf)

         if (recvd_type != NULL)
             *recvd_type = SSL3_RECORD_get_type(rr);
+        if (type == SSL3_RT_APPLICATION_DATA) {
+             /* mark any zero length record as consumed */
+             if (SSL3_RECORD_get_length(rr) == 0)
                  SSL3_RECORD_set_read(rr);

         if (len <= 0)
            return (len);

Thanks.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mta.openssl.org/pipermail/openssl-dev/attachments/20171102/e9ad62e0/attachment.html>