[openssl-dev] Memory leak in application when we use ECDH

Richard Levitte levitte at openssl.org
Mon Mar 27 08:50:29 UTC 2017


Are you confident this is the same leak that you have experienced?
Your original complaint was about keys created with the callback
that's added with SSL_set_tmp_ecdh_callback(), and I have to point out
that s_server doesn't provide such a callback.  In fact,
SSL_set_tmp_ecdh_callback() isn't called anywhere in the OpenSSL
source.

That doesn't mean you haven't found the same leak, if there is a leak
(I'm not going to rule out some other explanation), it might simply
clarify that targeting on the EC_KEY returned from the callback might
make us blind for something else that's related.

Cheers,
Richard

In message <25D2EC755404B4409F263AC6D050FEBB2A108EFC at AZ-FFEXMB03.global.avaya.com> on Mon, 27 Mar 2017 08:40:30 +0000, "Mody, Darshan (Darshan)" <darshanmody at avaya.com> said:

darshanmody> Matt, Richard,
darshanmody> 
darshanmody> I have not created any patch. Instead I am using s_server app provided from Openssl. Below is the server side command run
darshanmody> 
darshanmody> openssl s_server -accept <port> -key <custom-key> -cert <custom-certificate signed by custom CA> -CAfile <custom self signed CA>  -verify <custom self signed CA>
darshanmody> 
darshanmody> On the client side I am running script below
darshanmody> 
darshanmody> #/bin/sh
darshanmody> 
darshanmody> while [ 1 ]
darshanmody> do
darshanmody>      openssl s_client -connect <server-ip>:<server-port> -cert <custom client certificate not trusted by server> -key <custom client certificate key> &>/dev/null
darshanmody> done
darshanmody> 
darshanmody> Running the client once and printing the results showed the cipher selected was ECDHE cipher.  I believe there is inherent leak in the ECDHE side of openssl. 
darshanmody> 
darshanmody> I find that the server started leaking was leaking. Check the Resident memory and the CPU. Resident memory increases after every 3 second. When I had stopped the client traffic the resident memory did not increase however it did not returned back to what it was earlier. Hope this will help to reproduce the issue.
darshanmody> 
darshanmody> top - 10:16:37 up 46 days,  1:29,  2 users,  load average: 0.03, 0.06, 0.08
darshanmody> Tasks:   1 total,   0 running,   1 sleeping,   0 stopped,   0 zombie
darshanmody> Cpu(s):  0.6%us,  0.1%sy,  0.0%ni, 99.3%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
darshanmody> Mem:   3924288k total,  3757716k used,   166572k free,    78200k buffers
darshanmody> Swap:  8388604k total,   999556k used,  7389048k free,   219736k cached
darshanmody> 
darshanmody>   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
darshanmody> 27303 root      20   0 39908 3528 2740 S  0.0  0.1   0:00.00 openssl
darshanmody> 
darshanmody> 
darshanmody> top - 10:16:40 up 46 days,  1:29,  2 users,  load average: 0.03, 0.06, 0.08
darshanmody> Tasks:   1 total,   1 running,   0 sleeping,   0 stopped,   0 zombie
darshanmody> Cpu(s):  1.6%us,  0.9%sy,  0.0%ni, 97.4%id,  0.0%wa,  0.0%hi,  0.1%si,  0.0%st
darshanmody> Mem:   3924288k total,  3757856k used,   166432k free,    78208k buffers
darshanmody> Swap:  8388604k total,   999556k used,  7389048k free,   219744k cached
darshanmody> 
darshanmody>   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
darshanmody> 27303 root      20   0 39908 3532 2740 R  4.3  0.1   0:00.13 openssl
darshanmody> 
darshanmody> 
darshanmody> top - 10:16:43 up 46 days,  1:29,  2 users,  load average: 0.03, 0.06, 0.08
darshanmody> Tasks:   1 total,   0 running,   1 sleeping,   0 stopped,   0 zombie
darshanmody> Cpu(s):  2.6%us,  0.8%sy,  0.1%ni, 96.2%id,  0.0%wa,  0.0%hi,  0.3%si,  0.0%st
darshanmody> Mem:   3924288k total,  3757980k used,   166308k free,    78232k buffers
darshanmody> Swap:  8388604k total,   999556k used,  7389048k free,   219744k cached
darshanmody> 
darshanmody>   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
darshanmody> 27303 root      20   0 39908 3544 2740 S  7.0  0.1   0:00.34 openssl
darshanmody> 
darshanmody> 
darshanmody> top - 10:16:46 up 46 days,  1:30,  2 users,  load average: 0.03, 0.06, 0.08
darshanmody> Tasks:   1 total,   0 running,   1 sleeping,   0 stopped,   0 zombie
darshanmody> Cpu(s):  2.1%us,  1.1%sy,  0.0%ni, 96.5%id,  0.0%wa,  0.0%hi,  0.2%si,  0.0%st
darshanmody> Mem:   3924288k total,  3757964k used,   166324k free,    78248k buffers
darshanmody> Swap:  8388604k total,   999556k used,  7389048k free,   219732k cached
darshanmody> 
darshanmody>   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
darshanmody> 27303 root      20   0 39908 3560 2740 S  6.7  0.1   0:00.54 openssl
darshanmody> 
darshanmody> 
darshanmody> top - 10:16:49 up 46 days,  1:30,  2 users,  load average: 0.02, 0.06, 0.08
darshanmody> Tasks:   1 total,   0 running,   1 sleeping,   0 stopped,   0 zombie
darshanmody> Cpu(s):  2.7%us,  1.1%sy,  0.0%ni, 96.0%id,  0.0%wa,  0.0%hi,  0.2%si,  0.0%st
darshanmody> Mem:   3924288k total,  3758212k used,   166076k free,    78264k buffers
darshanmody> Swap:  8388604k total,   999556k used,  7389048k free,   219744k cached
darshanmody> 
darshanmody>   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
darshanmody> 27303 root      20   0 39908 3572 2740 S  8.0  0.1   0:00.78 openssl
darshanmody> 
darshanmody> 
darshanmody> top - 10:16:52 up 46 days,  1:30,  2 users,  load average: 0.02, 0.06, 0.08
darshanmody> Tasks:   1 total,   0 running,   1 sleeping,   0 stopped,   0 zombie
darshanmody> Cpu(s):  3.2%us,  1.2%sy,  0.0%ni, 94.8%id,  0.1%wa,  0.0%hi,  0.6%si,  0.0%st
darshanmody> Mem:   3924288k total,  3758212k used,   166076k free,    78288k buffers
darshanmody> Swap:  8388604k total,   999556k used,  7389048k free,   219732k cached
darshanmody> 
darshanmody>   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
darshanmody> 27303 root      20   0 39908 3592 2740 S 11.0  0.1   0:01.11 openssl
darshanmody> 
darshanmody> 
darshanmody> top - 10:16:55 up 46 days,  1:30,  2 users,  load average: 0.02, 0.06, 0.08
darshanmody> Tasks:   1 total,   0 running,   1 sleeping,   0 stopped,   0 zombie
darshanmody> Cpu(s):  3.3%us,  1.3%sy,  0.1%ni, 94.7%id,  0.0%wa,  0.0%hi,  0.6%si,  0.0%st
darshanmody> Mem:   3924288k total,  3758584k used,   165704k free,    78288k buffers
darshanmody> Swap:  8388604k total,   999556k used,  7389048k free,   219744k cached
darshanmody> 
darshanmody>   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
darshanmody> 27303 root      20   0 39908 3608 2740 S 11.3  0.1   0:01.45 openssl
darshanmody> 
darshanmody> 
darshanmody> top - 10:16:58 up 46 days,  1:30,  2 users,  load average: 0.02, 0.06, 0.08
darshanmody> Tasks:   1 total,   0 running,   1 sleeping,   0 stopped,   0 zombie
darshanmody> Cpu(s):  3.3%us,  1.6%sy,  0.0%ni, 94.7%id,  0.0%wa,  0.0%hi,  0.4%si,  0.0%st
darshanmody> Mem:   3924288k total,  3758864k used,   165424k free,    78296k buffers
darshanmody> Swap:  8388604k total,   999556k used,  7389048k free,   219744k cached
darshanmody> 
darshanmody>   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
darshanmody> 27303 root      20   0 39908 3628 2740 S 11.3  0.1   0:01.79 openssl
darshanmody> 
darshanmody> 
darshanmody> top - 10:17:01 up 46 days,  1:30,  2 users,  load average: 0.02, 0.06, 0.08
darshanmody> Tasks:   1 total,   0 running,   1 sleeping,   0 stopped,   0 zombie
darshanmody> Cpu(s):  3.4%us,  1.5%sy,  0.0%ni, 94.7%id,  0.0%wa,  0.0%hi,  0.4%si,  0.0%st
darshanmody> Mem:   3924288k total,  3758956k used,   165332k free,    78312k buffers
darshanmody> Swap:  8388604k total,   999556k used,  7389048k free,   219740k cached
darshanmody> 
darshanmody>   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
darshanmody> 27303 root      20   0 40016 3648 2740 S 10.7  0.1   0:02.11 openssl
darshanmody> 
darshanmody> 
darshanmody> top - 10:17:04 up 46 days,  1:30,  2 users,  load average: 0.02, 0.06, 0.07
darshanmody> Tasks:   1 total,   0 running,   1 sleeping,   0 stopped,   0 zombie
darshanmody> Cpu(s):  3.9%us,  1.9%sy,  0.0%ni, 91.9%id,  1.6%wa,  0.0%hi,  0.7%si,  0.0%st
darshanmody> Mem:   3924288k total,  3759708k used,   164580k free,    78328k buffers
darshanmody> Swap:  8388604k total,   999556k used,  7389048k free,   219916k cached
darshanmody> 
darshanmody>   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
darshanmody> 27303 root      20   0 40016 3668 2740 S 11.0  0.1   0:02.44 openssl
darshanmody> 
darshanmody> 
darshanmody> top - 10:17:07 up 46 days,  1:30,  2 users,  load average: 0.02, 0.06, 0.07
darshanmody> Tasks:   1 total,   1 running,   0 sleeping,   0 stopped,   0 zombie
darshanmody> Cpu(s):  3.4%us,  1.2%sy,  0.1%ni, 93.8%id,  0.0%wa,  0.0%hi,  1.4%si,  0.0%st
darshanmody> Mem:   3924288k total,  3759336k used,   164952k free,    78360k buffers
darshanmody> Swap:  8388604k total,   999556k used,  7389048k free,   219900k cached
darshanmody> 
darshanmody>   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
darshanmody> 27303 root      20   0 40016 3688 2740 R 10.7  0.1   0:02.76 openssl
darshanmody> 
darshanmody> 
darshanmody> top - 10:17:10 up 46 days,  1:30,  2 users,  load average: 0.02, 0.06, 0.07
darshanmody> Tasks:   1 total,   0 running,   1 sleeping,   0 stopped,   0 zombie
darshanmody> Cpu(s):  3.2%us,  1.6%sy,  0.0%ni, 94.2%id,  0.0%wa,  0.0%hi,  1.0%si,  0.0%st
darshanmody> Mem:   3924288k total,  3758584k used,   165704k free,    78360k buffers
darshanmody> Swap:  8388604k total,   999556k used,  7389048k free,   219916k cached
darshanmody> 
darshanmody>   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
darshanmody> 27303 root      20   0 40016 3708 2740 S 11.3  0.1   0:03.10 openssl
darshanmody> 
darshanmody> 
darshanmody> top - 10:17:13 up 46 days,  1:30,  2 users,  load average: 0.02, 0.06, 0.07
darshanmody> Tasks:   1 total,   0 running,   1 sleeping,   0 stopped,   0 zombie
darshanmody> Cpu(s):  3.6%us,  1.1%sy,  0.0%ni, 94.5%id,  0.0%wa,  0.0%hi,  0.8%si,  0.0%st
darshanmody> Mem:   3924288k total,  3758476k used,   165812k free,    78376k buffers
darshanmody> Swap:  8388604k total,   999556k used,  7389048k free,   219920k cached
darshanmody> 
darshanmody>   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
darshanmody> 27303 root      20   0 40016 3728 2740 S 10.7  0.1   0:03.42 openssl
darshanmody> 
darshanmody> 
darshanmody> top - 10:17:16 up 46 days,  1:30,  2 users,  load average: 0.02, 0.06, 0.07
darshanmody> Tasks:   1 total,   0 running,   1 sleeping,   0 stopped,   0 zombie
darshanmody> Cpu(s):  3.8%us,  1.3%sy,  0.0%ni, 93.3%id,  0.0%wa,  0.0%hi,  1.7%si,  0.0%st
darshanmody> Mem:   3924288k total,  3759824k used,   164464k free,    78384k buffers
darshanmody> Swap:  8388604k total,   999556k used,  7389048k free,   219920k cached
darshanmody> 
darshanmody>   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
darshanmody> 27303 root      20   0 40124 3748 2740 S 11.3  0.1   0:03.76 openssl
darshanmody> 
darshanmody> ---------------
darshanmody> 
darshanmody> top - 10:23:10 up 46 days,  1:36,  2 users,  load average: 0.03, 0.05, 0.06
darshanmody> Tasks:   1 total,   0 running,   1 sleeping,   0 stopped,   0 zombie
darshanmody> Cpu(s):  3.6%us,  1.3%sy,  0.1%ni, 94.4%id,  0.0%wa,  0.0%hi,  0.6%si,  0.0%st
darshanmody> Mem:   3924288k total,  3756048k used,   168240k free,    75888k buffers
darshanmody> Swap:  8388604k total,   998972k used,  7389632k free,   216576k cached
darshanmody> 
darshanmody>   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
darshanmody> 27303 root      20   0 42392 6048 2740 S 10.7  0.2   0:41.95 openssl
darshanmody> 
darshanmody> 
darshanmody> top - 10:23:13 up 46 days,  1:36,  2 users,  load average: 0.02, 0.04, 0.06
darshanmody> Tasks:   1 total,   0 running,   1 sleeping,   0 stopped,   0 zombie
darshanmody> Cpu(s):  3.2%us,  1.3%sy,  0.0%ni, 94.8%id,  0.0%wa,  0.0%hi,  0.6%si,  0.0%st
darshanmody> Mem:   3924288k total,  3756064k used,   168224k free,    75904k buffers
darshanmody> Swap:  8388604k total,   998972k used,  7389632k free,   216572k cached
darshanmody> 
darshanmody>   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
darshanmody> 27303 root      20   0 42392 6068 2740 S 10.7  0.2   0:42.27 openssl
darshanmody> 
darshanmody> 
darshanmody> top - 10:23:16 up 46 days,  1:36,  2 users,  load average: 0.02, 0.04, 0.06
darshanmody> Tasks:   1 total,   1 running,   0 sleeping,   0 stopped,   0 zombie
darshanmody> Cpu(s):  3.2%us,  1.3%sy,  0.1%ni, 94.9%id,  0.0%wa,  0.0%hi,  0.4%si,  0.0%st
darshanmody> Mem:   3924288k total,  3756188k used,   168100k free,    75912k buffers
darshanmody> Swap:  8388604k total,   998972k used,  7389632k free,   216572k cached
darshanmody> 
darshanmody>   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
darshanmody> 27303 root      20   0 42392 6088 2740 R 11.0  0.2   0:42.60 openssl
darshanmody> 
darshanmody> 
darshanmody> top - 10:23:19 up 46 days,  1:36,  2 users,  load average: 0.02, 0.04, 0.06
darshanmody> Tasks:   1 total,   0 running,   1 sleeping,   0 stopped,   0 zombie
darshanmody> Cpu(s):  3.1%us,  1.2%sy,  1.6%ni, 93.5%id,  0.0%wa,  0.0%hi,  0.6%si,  0.0%st
darshanmody> Mem:   3924288k total,  3756528k used,   167760k free,    75920k buffers
darshanmody> Swap:  8388604k total,   998972k used,  7389632k free,   216584k cached
darshanmody> 
darshanmody>   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
darshanmody> 27303 root      20   0 42392 6108 2740 S 10.7  0.2   0:42.92 openssl
darshanmody> 
darshanmody> 
darshanmody> top - 10:23:22 up 46 days,  1:36,  2 users,  load average: 0.02, 0.04, 0.06
darshanmody> Tasks:   1 total,   0 running,   1 sleeping,   0 stopped,   0 zombie
darshanmody> Cpu(s): 11.1%us,  4.6%sy, 20.2%ni, 63.6%id,  0.0%wa,  0.0%hi,  0.6%si,  0.0%st
darshanmody> Mem:   3924288k total,  3756924k used,   167364k free,    75952k buffers
darshanmody> Swap:  8388604k total,   998972k used,  7389632k free,   216600k cached
darshanmody> 
darshanmody>   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
darshanmody> 27303 root      20   0 42500 6124 2740 S 10.3  0.2   0:43.23 openssl
darshanmody> 
darshanmody> Thanks
darshanmody> Darshan
darshanmody> 
darshanmody> -----Original Message-----
darshanmody> From: openssl-dev [mailto:openssl-dev-bounces at openssl.org] On Behalf Of Mody, Darshan (Darshan)
darshanmody> Sent: Thursday, March 23, 2017 7:40 PM
darshanmody> To: openssl-dev at openssl.org
darshanmody> Cc: Bahr, William G (Bill)
darshanmody> Subject: Re: [openssl-dev] Memory leak in application when we use ECDH
darshanmody> 
darshanmody> Thanks Richard and Matt,
darshanmody> 
darshanmody> I will patch it and send the patch. It will take me couple of days.
darshanmody> 
darshanmody> Regards
darshanmody> Darshan
darshanmody> 
darshanmody> -----Original Message-----
darshanmody> From: openssl-dev [mailto:openssl-dev-bounces at openssl.org] On Behalf Of Richard Levitte
darshanmody> Sent: Thursday, March 23, 2017 7:31 PM
darshanmody> To: openssl-dev at openssl.org
darshanmody> Subject: Re: [openssl-dev] Memory leak in application when we use ECDH
darshanmody> 
darshanmody> I think that Matt is asking for example code that exhibits this leak.
darshanmody> You could patch apps/s_server.c with your callback, or ssl/ssltest.c, and give us that patch.
darshanmody> 
darshanmody> The reason is that we can't know what assumptions you're going with in your callback or application, so if we code an example together, it will be with Our conditions, not yours, and therefore a pretty bad method to figure this out.
darshanmody> 
darshanmody> Cheers,
darshanmody> Richard
darshanmody> 
darshanmody> In message <25D2EC755404B4409F263AC6D050FEBB2A10781E at AZ-FFEXMB03.global.avaya.com> on Thu, 23 Mar 2017 13:47:10 +0000, "Mody, Darshan (Darshan)" <darshanmody at avaya.com> said:
darshanmody> 
darshanmody> darshanmody> Matt,
darshanmody> darshanmody> 
darshanmody> darshanmody> Below is the scenario.
darshanmody> darshanmody> 
darshanmody> darshanmody> 1. Have server open a listen socket which always validates the client certificate and chain.
darshanmody> darshanmody> 2. On server support ECDHE using callback. Ensure the EC_KEY passed to openssl from app is cleaned up by the app.
darshanmody> darshanmody> 3. Connect client with certificates that server does not trust.
darshanmody> darshanmody> 4. The connections from client to server fails
darshanmody> darshanmody> 
darshanmody> darshanmody> In course of time the app running the server has been leaking. Even after accounting for the EC_KEY passed by the server app to openssl we find there seems to be leak. Further investigation on the core dumps generated from the server app shows that it has the certificates from the client saved.
darshanmody> darshanmody> 
darshanmody> darshanmody> Hope this helps
darshanmody> darshanmody> 
darshanmody> darshanmody> Thanks
darshanmody> darshanmody> Darshan
darshanmody> darshanmody> 
darshanmody> darshanmody> -----Original Message-----
darshanmody> darshanmody> From: openssl-dev [mailto:openssl-dev-bounces at openssl.org]
darshanmody> darshanmody> On Behalf Of Matt Caswell
darshanmody> darshanmody> Sent: Thursday, March 23, 2017 6:55 PM
darshanmody> darshanmody> To: openssl-dev at openssl.org
darshanmody> darshanmody> Subject: Re: [openssl-dev] Memory leak in application when 
darshanmody> darshanmody> we use ECDH
darshanmody> darshanmody> 
darshanmody> darshanmody> 
darshanmody> darshanmody> 
darshanmody> darshanmody> On 23/03/17 13:19, Mody, Darshan (Darshan) wrote:
darshanmody> darshanmody> > Can you further elaborate?
darshanmody> darshanmody> > 
darshanmody> darshanmody> > What we did is to create a TLS connection and with 
darshanmody> darshanmody> > invalid certificates from the client and server on 
darshanmody> darshanmody> > verification would reject the certificate. The cipher 
darshanmody> darshanmody> > negotiated was ECDHE cipher between client and server.
darshanmody> darshanmody> > 
darshanmody> darshanmody> > This was done with load (multiple while 1 script trying 
darshanmody> darshanmody> > to connect to server using invalid certificates and in 
darshanmody> darshanmody> > course of time the memory was increasing).
darshanmody> darshanmody> 
darshanmody> darshanmody> Without being able to recreate the problem its going to be very difficult/impossible for us to fix it (assuming the problem is in OpenSSl itself). We would need some simple reproducer code that demonstrates the problem occurring.
darshanmody> darshanmody> 
darshanmody> darshanmody> Matt
darshanmody> darshanmody> 
darshanmody> darshanmody> 
darshanmody> darshanmody> > 
darshanmody> darshanmody> > Thanks Darshan
darshanmody> darshanmody> > 
darshanmody> darshanmody> > -----Original Message----- From: openssl-dev 
darshanmody> darshanmody> > [mailto:openssl-dev-bounces at openssl.org] On Behalf Of 
darshanmody> darshanmody> > Matt Caswell
darshanmody> darshanmody> > Sent: Thursday, March 23, 2017 4:09 PM To: 
darshanmody> darshanmody> > openssl-dev at openssl.org
darshanmody> darshanmody> > Subject: Re: [openssl-dev] Memory leak in application 
darshanmody> darshanmody> > when we use ECDH
darshanmody> darshanmody> > 
darshanmody> darshanmody> > 
darshanmody> darshanmody> > 
darshanmody> darshanmody> > On 23/03/17 10:13, Mody, Darshan (Darshan) wrote:
darshanmody> darshanmody> >> Matt,
darshanmody> darshanmody> >> 
darshanmody> darshanmody> >> Even after accounting for the EC_KEY we still observe some leak.
darshanmody> darshanmody> >> The leak started after we started using supporting EC 
darshanmody> darshanmody> >> with callback SSL_set_tmp_ecdh_callback().
darshanmody> darshanmody> >> 
darshanmody> darshanmody> >> The core dump shows  the string data of the far-end certificates.
darshanmody> darshanmody> >> I cannot pin point  the code in openssl with this regard.
darshanmody> darshanmody> > 
darshanmody> darshanmody> > Are you able to create a simple reproducer demonstrating 
darshanmody> darshanmody> > the problem with the callback?
darshanmody> darshanmody> > 
darshanmody> darshanmody> > Matt
darshanmody> darshanmody> > 
darshanmody> darshanmody> --
darshanmody> darshanmody> openssl-dev mailing list
darshanmody> darshanmody> To unsubscribe: 
darshanmody> darshanmody> https://urldefense.proofpoint.com/v2/url?u=https-3A__mta.op
darshanmody> darshanmody> enssl.org_mailman_listinfo_openssl-2Ddev&d=DwICAg&c=BFpWQw8
darshanmody> darshanmody> bsuKpl1SgiZH64Q&r=bsEULbVnjelD7InzgsegHBEbtXzaIDagy9EuEhJrK
darshanmody> darshanmody> fQ&m=VbrRgO8PZIVkFM4PjeK7TEgKDHnbLu_QfbyqRhmvx8I&s=u0cR7sQf
darshanmody> darshanmody> _Zz8FoCnrzgLc3drBSR8Ou1qDUyxV8z1xYQ&e=
darshanmody> darshanmody> --
darshanmody> darshanmody> openssl-dev mailing list
darshanmody> darshanmody> To unsubscribe: 
darshanmody> darshanmody> https://urldefense.proofpoint.com/v2/url?u=https-3A__mta.op
darshanmody> darshanmody> enssl.org_mailman_listinfo_openssl-2Ddev&d=DwICAg&c=BFpWQw8
darshanmody> darshanmody> bsuKpl1SgiZH64Q&r=bsEULbVnjelD7InzgsegHBEbtXzaIDagy9EuEhJrK
darshanmody> darshanmody> fQ&m=OtZlUFiavvOVqXL900IST85y3pZLikUdEgekBIIyZoI&s=3T5xlm8q
darshanmody> darshanmody> 92-eP1ItbDzGOU972l4wFrkJUgLrBNR4Qx8&e=
darshanmody> darshanmody> 
darshanmody> --
darshanmody> openssl-dev mailing list
darshanmody> To unsubscribe: https://urldefense.proofpoint.com/v2/url?u=https-3A__mta.openssl.org_mailman_listinfo_openssl-2Ddev&d=DwICAg&c=BFpWQw8bsuKpl1SgiZH64Q&r=bsEULbVnjelD7InzgsegHBEbtXzaIDagy9EuEhJrKfQ&m=OtZlUFiavvOVqXL900IST85y3pZLikUdEgekBIIyZoI&s=3T5xlm8q92-eP1ItbDzGOU972l4wFrkJUgLrBNR4Qx8&e=
darshanmody> --
darshanmody> openssl-dev mailing list
darshanmody> To unsubscribe: https://urldefense.proofpoint.com/v2/url?u=https-3A__mta.openssl.org_mailman_listinfo_openssl-2Ddev&d=DwICAg&c=BFpWQw8bsuKpl1SgiZH64Q&r=bsEULbVnjelD7InzgsegHBEbtXzaIDagy9EuEhJrKfQ&m=aM8Nc6bSJzLs-HdERDikYMeQK6uUqvjYvWnfqtvREM4&s=5DECvTKmFHf9urfCwMDaDnk14k9fM_XDkpl9X5r5aJA&e= 
darshanmody> -- 
darshanmody> openssl-dev mailing list
darshanmody> To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev
darshanmody> 


More information about the openssl-dev mailing list