[openssl-users] OpenSSL performance issue

Prabhat Puroshottam prabhat.puroshottam at outlook.com
Thu Dec 18 21:00:07 UTC 2014


Hi,

I think my last email was somehow lost in transition between the mail
servers so I am starting afresh if somebody can help. If you have
already taken pains to read through this mail, kindly skip to the bottom
of the mail. Thanks for your patience.

 
First let me state upfront that I am relatively very new to OpenSSL.
Also please forgive me if this is not the correct mailing list  for this issue.
 
We have a product which uses OpenSSL to connect and transfer 
application level data. There are two ways to connect, and get the application
level data from *Agent* to *Client*
 
1. Client (C/C++) -> Agent (C/C++)
2. Client (C/C++) -> Proxy Server (Java) -> Agent (C/C++)
 
*Client* and *Agent* are implemented in C, while *Proxy Server* uses Java code 
(This shouldn't really matter). But might be helpful for you to know.
The issue is, connecting *Client* to *Agent* is very fast (that is relatively). 
While connecting *Client* to *Proxy Server* is very slow - that is orders of 
magnitudes slow.
 
I was trying to determine the root cause. From my analysis is appears that, 
maximum time on the *Client* side is taken by SSL_Connect during connection 
establishment, while the actual application level data transfer takes very 
small time. Similarly, on the *Proxy Server* side (Java Code), maximum time 
is taken in the first read/write whichever happens first. Further, I don't 
think this is a network latency issue, as the problem is very pronounced 
and all the three boxes are on the same network. Also, the *Client* code 
seems to be similar, whether we connect to *Agent* (method 1 above) or 
*Proxy Server* (method 2 above). So, the issue is with *Proxy Server*, IMHO.
 
 
To further locate the issue, I did some tests using ssldump command,
 
***************************************
This is for *Client -> Agent*
***************************************
 
1 1  0.0023 (0.0002)  C>S  Handshake
      ClientHello
        Version 3.1
        cipher suites
        TLS_DHE_RSA_WITH_AES_256_CBC_SHA
        TLS_DHE_DSS_WITH_AES_256_CBC_SHA
        TLS_RSA_WITH_AES_256_CBC_SHA
        TLS_DHE_RSA_WITH_3DES_EDE_CBC_SHA
        TLS_DHE_DSS_WITH_3DES_EDE_CBC_SHA
        TLS_RSA_WITH_3DES_EDE_CBC_SHA
        TLS_DHE_RSA_WITH_AES_128_CBC_SHA
        TLS_DHE_DSS_WITH_AES_128_CBC_SHA
        TLS_RSA_WITH_AES_128_CBC_SHA
        TLS_RSA_WITH_IDEA_CBC_SHA
        TLS_RSA_WITH_RC4_128_SHA
        Unknown value 0xff
        compression methods
                  NULL
1 2  0.0091 (0.0068)  S>C  Handshake
      ServerHello
        Version 3.1
        session_id[0]=
 
        cipherSuite         TLS_RSA_WITH_AES_256_CBC_SHA
        compressionMethod                   NULL
1 3  0.0091 (0.0000)  S>C  Handshake
      Certificate
1 4  0.0091 (0.0000)  S>C  Handshake
      CertificateRequest
        certificate_types                   rsa_sign
        certificate_types                   dss_sign
      ServerHelloDone
1 5  0.0104 (0.0013)  C>S  Handshake
      Certificate
1 6  0.0104 (0.0000)  C>S  Handshake
      ClientKeyExchange
1 7  0.0104 (0.0000)  C>S  ChangeCipherSpec
1 8  0.0104 (0.0000)  C>S  Handshake
1 9  0.0159 (0.0054)  S>C  Handshake
      TLS_RSA_WITH_RC4_128_MD51 10 0.0159 (0.0000)  S>C  ChangeCipherSpec
1 11 0.0159 (0.0000)  S>C  Handshake
1 12 0.0163 (0.0004)  C>S  application_data
1 13 0.0174 (0.0011)  S>C  application_data
1 14 0.0176 (0.0001)  C>S  application_data
1 15 0.0186 (0.0010)  S>C  application_data
1 16 0.0189 (0.0002)  C>S  application_data
1 17 0.0200 (0.0010)  S>C  application_data
1 18 0.0203 (0.0002)  C>S  application_data
1    0.0205 (0.0002)  C>S  TCP FIN
1 19 0.0207 (0.0001)  S>C  Alert
1    0.0209 (0.0001)  S>C  TCP FIN
 
 
***************************************
This is for *Client -> Proxy Server*
***************************************
 
1 1  0.0025 (0.0025)  C>S  Handshake
      ClientHello
        Version 3.1
        cipher suites
        TLS_DHE_RSA_WITH_AES_256_CBC_SHA
        TLS_DHE_DSS_WITH_AES_256_CBC_SHA
        TLS_RSA_WITH_AES_256_CBC_SHA
        TLS_DHE_RSA_WITH_3DES_EDE_CBC_SHA
        TLS_DHE_DSS_WITH_3DES_EDE_CBC_SHA
        TLS_RSA_WITH_3DES_EDE_CBC_SHA
        TLS_DHE_RSA_WITH_AES_128_CBC_SHA
        TLS_DHE_DSS_WITH_AES_128_CBC_SHA
        TLS_RSA_WITH_AES_128_CBC_SHA
        TLS_RSA_WITH_IDEA_CBC_SHA
        TLS_RSA_WITH_RC4_128_SHA
        Unknown value 0xff
        compression methods
                  NULL
1 2  3.2949 (3.2923)  S>C  Handshake
      ServerHello
        Version 3.1
        session_id[32]=
          54 76 ca 2c bd 84 37 f9 98 7a b1 57 1f 61 9a 3d
          40 89 58 89 e6 14 5f 39 8b 96 62 87 8f a9 30 8f
        cipherSuite         TLS_DHE_RSA_WITH_AES_256_CBC_SHA
        compressionMethod                   NULL
      Certificate
      ServerKeyExchange
      ServerHelloDone
1 3  3.3135 (0.0186)  C>S  Handshake
      ClientKeyExchange
        DiffieHellmanClientPublicValue[96]=
          9c 38 d3 b5 10 3f 4d a6 54 a9 84 02 b3 f2 25 79
          e1 f0 6e 6f 56 35 44 73 13 40 4e c0 6b 2a e7 64
          ff 4a a0 7a 34 82 1e d9 35 70 d1 43 f8 c5 18 c4
          56 54 a6 32 33 bc 1a 35 62 fe 36 cc 33 f4 a0 15
          6d 7a ea 5e a6 f5 d6 d7 21 45 16 30 08 2b bc e9
          4c 25 32 64 b3 a3 b9 21 db 26 b1 6c 4c 92 34 aa
1 4  3.3135 (0.0000)  C>S  ChangeCipherSpec
1 5  3.3135 (0.0000)  C>S  Handshake
1 6  3.3241 (0.0106)  S>C  ChangeCipherSpec
1 7  3.3638 (0.0396)  S>C  Handshake
1 8  3.3638 (0.0000)  S>C  application_data
1 9  3.3646 (0.0007)  C>S  application_data
1 10 3.3646 (0.0000)  C>S  application_data
1 11 3.4196 (0.0549)  S>C  application_data
1 12 3.4196 (0.0000)  S>C  application_data
1 13 3.4270 (0.0073)  C>S  application_data
1 14 3.4271 (0.0000)  C>S  application_data
1 15 3.4271 (0.0000)  C>S  application_data
1 16 3.4646 (0.0375)  S>C  application_data
1 17 3.4646 (0.0000)  S>C  application_data
1 18 3.4649 (0.0002)  C>S  application_data
1 19 3.4649 (0.0000)  C>S  application_data
1    3.4652 (0.0003)  C>S  TCP FIN
1 20 3.4657 (0.0005)  S>C  Alert
1    3.4658 (0.0000)  S>C  TCP FIN
 
************************************************************
 
As you can see the big time difference between the two executions - which 
actually involve the same application level data. The largest chunk of 
time is spent waiting for handshake from *Proxy Server*. The response time 
of *Proxy Server* in replying back with ServerHello, varies greatly 
between 1.5 to 11 seconds across different runs. In the present case it is 
nearly 3.3 seconds - which IMO is not acceptable.


*****************************************************
Above this was previous content.


> From: owner-openssl-users at openssl.org On Behalf Of Dave Thompson

> does Proxy code do between accepting on the [SSL]ServerSocket 
> (which is Java's way of representing the listen-ing socket) and the 
> first read or write -- apparently write? One particular thing, is it 
> using BufferedReader/Writer on the socket streams? Java code 
> often does buffered because it is more efficient than going to OS 
> each time (at least for real I/O rather than ByteArray Streams).
> In that case for output you need .flush() to actually send. 
> Or you could try doing .startHandshake() explicitly as soon as 
> convenient after the .accept() and see what difference that 
> makes (although it may only move the problem elsewhere).

Thanks Dave for your pointer. The code is actually using 
BufferedReader/Writer. Further, I actually tried adding 
startHandshake() explicitly after socket accept(). However, what I 
observed is that now startHandshake() gets blocked and takes up 
the same amount of time as the first read/write used to take. After
this change, the actual read/write calls become very fast including 
the first call. So the problem it appears is in the SSL handshake 
response code from the java side. 

I am pretty sure the issue is not related to OpenSSL but in the 
java side. Further, since it is startHandshake() that is blocking for this
much time, it is not a problem with *our* Java Code. But just to be 
sure could it have anything to do with ciphers/encryption/keys etc. 
Any other pointers/help is highly appreciated.


Thanks, for reading through such a lengthy email. If anybody can kindly 
provide his inputs, or even point me in the right direction, I shall be 
highly grateful. Any other comments or suggestions are also highly 
welcome. Thanks for your patience,
 
Prabhat.

 		 	   		  


More information about the openssl-users mailing list