<html>
<head>
<meta content="text/html; charset=UTF-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
Correct. Locks 39/40 are only useful while the POST is running.
Once the POST completes, the POST status never changes again unless
the POST runs again. The only way to run the POST is by invoking
FIPS_mode_set(1). But there should be no reason to invoke
FIPS_mode_set(1) more than once unless you want to provide some sort
of run-time FIPS mode toggle capability. <br>
<br>
You're asking in the context of Apache, right? Apache just has a
global FIPS on/off knob in the config. If set, FIPS is enabled for
the lifetime of the process and never disabled/re-enabled. You would
invoke FIPS_mode_set(1) once prior to spinning up the worker
threads. Therefore, the POST would be finished prior to any
multi-threaded processing.<br>
<br>
<br>
<br>
<div class="moz-cite-prefix">On 04/29/2015 01:53 PM, Bryan Call
wrote:<br>
</div>
<blockquote
cite="mid:E98AE19B-413A-40D3-BC9C-67C83BBC3EB8@apache.org"
type="cite">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
Can I safely assume that if I call FIPS_mode_set(0) and get a
successful return value then I don’t need to lock when there are
callbacks for type 39 and 40 locks (for OpenSSL 1.0.1 and 1.0.2)?
<div class=""><br class="">
<div class="">
<div style="color: rgb(0, 0, 0); letter-spacing: normal;
orphans: auto; text-align: start; text-indent: 0px;
text-transform: none; white-space: normal; widows: auto;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
word-wrap: break-word; -webkit-nbsp-mode: space;
-webkit-line-break: after-white-space;" class="">
<div class="">-Bryan</div>
<div class=""><br class="">
</div>
</div>
<br class="Apple-interchange-newline">
<br class="Apple-interchange-newline">
</div>
<br class="">
<div>
<blockquote type="cite" class="">
<div class="">On Apr 28, 2015, at 10:22 AM, John Foley <<a
moz-do-not-send="true" href="mailto:foleyj@cisco.com"
class="">foleyj@cisco.com</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div bgcolor="#FFFFFF" text="#000000" class=""> In the
context of OpenSSL 1.0.1 or 1.0.2, this means
FIPS_mode_set(1) has returned with a successful return
value. <br class="">
<br class="">
In the future, this may change if/when OpenSSL does
another FIPS validation. CMVP has changed the
implementation guidance that requires the POST to run
automatically without user intervention. Any future
FIPS validations would be subject to the new rules.
Hence, the behavior of FIPS_mode_set() may change in the
future if OpenSSL decides to pursue another validation.<br
class="">
<br class="">
<br class="">
<br class="">
<div class="moz-cite-prefix">On 04/28/2015 12:43 PM,
Bryan Call wrote:<br class="">
</div>
<blockquote
cite="mid:08D500E9-5A78-4CB8-B49B-6F41523A2C2E@apache.org"
type="cite" class=""> What do you mean by “FIPS POST
has completed”?
<div class=""><br class="">
<div class="">
<div style="letter-spacing: normal; orphans: auto;
text-align: start; text-indent: 0px;
text-transform: none; white-space: normal;
widows: auto; word-spacing: 0px;
-webkit-text-stroke-width: 0px; word-wrap:
break-word; -webkit-nbsp-mode: space;
-webkit-line-break: after-white-space;" class="">
<div class="">-Bryan</div>
<div class=""><br class="">
</div>
</div>
<br class="Apple-interchange-newline">
<br class="Apple-interchange-newline">
</div>
<br class="">
<div class="">
<blockquote type="cite" class="">
<div class="">On Apr 24, 2015, at 4:17 PM, John
Foley <<a moz-do-not-send="true"
href="mailto:foleyj@cisco.com" class="">foleyj@cisco.com</a>>
wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div bgcolor="#FFFFFF" text="#000000" class="">
Some of the algorithms still invoke the FIPS
flavor of the algorithm even when FIPS is
disabled. For example, this code is from
EVP_DigestUpdate().<br class="">
<br class="">
int EVP_DigestUpdate(EVP_MD_CTX *ctx, const
void *data, size_t count)<br class="">
{<br class="">
#ifdef OPENSSL_FIPS<br class="">
return FIPS_digestupdate(ctx, data,
count);<br class="">
#else<br class="">
return ctx->update(ctx, data, count);<br
class="">
#endif<br class="">
}<br class="">
<br class="">
In 1.0.1, locks 39 and 40 are:<br class="">
<br class="">
# define CRYPTO_LOCK_FIPS 39<br
class="">
# define CRYPTO_LOCK_FIPS2 40<br
class="">
<br class="">
You should try adding some logic to skip the
lock in your lock handler when the lock ID
is 39 or 40. Again, it should be safe to not
lock on these two as long as the FIPS POST
has completed.<br class="">
<br class="">
<br class="">
<br class="">
<div class="moz-cite-prefix">On 04/24/2015
05:56 PM, Bryan Call wrote:<br class="">
</div>
<blockquote
cite="mid:A5F20DF3-E379-4952-82BB-C3CB3A4BC5DA@apache.org"
type="cite" class=""> In my last email I
ran the benchmark on Fedora 21 (big
mistake). Here are the results when
running it back on the 28/56 core RHEL 6.5
server showing contention in a different
place (fips). Is there a reason it would
be calling into the FIPS code to get a
lock of FIPS is not enabled?
<div class="">
<div class=""><br class="">
</div>
</div>
<div class="">Contention log:</div>
<div class="">
<div class="">[Apr 24 21:35:09.731]
Server {0x7f5529ccd700} ERROR:
contention for lock - total
contention: 70662 waiting: 16 file:
fips.c line: 471 type: 40</div>
<div class="">[Apr 24 21:35:09.732]
Server {0x7f55299ca700} ERROR:
contention for lock - total
contention: 71605 waiting: 2 file:
fips.c line: 107 type: 39</div>
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class="">Line in in fips that see the
contention the most:</div>
<div class="">
<div class="">[bcall@bm77
trafficserver]$ grep fips.c diags.log
| cut -f 19 -d ' ' | sort | uniq -c |
sort -rn</div>
<div class=""> 875379 471</div>
<div class=""> 288338 107</div>
<div class=""> 275472 127</div>
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class="">Different lock types and
their contention frequency (multiple
benchmarks):</div>
<div class="">
<div class=""> 875379 40 < - in
fips.c</div>
<div class=""> 563810 39 <- also in
fips.c</div>
<div class=""> 440518 1</div>
<div class=""> 44145 18</div>
<div class=""> 17447 2</div>
<div class=""> 11356 10</div>
<div class=""> 1377 19</div>
<div class=""> 687 12</div>
<div class=""> 532 25</div>
<div class=""> 302 33</div>
<div class=""> 239 9</div>
<div class=""> 58 16</div>
<div class=""> 40 21</div>
<div class=""> 32 11</div>
<div class=""> 19 3</div>
<div class=""> 1 14</div>
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class="">Code to diable FIPS and log:</div>
<div class="">
<div class="">
<div class="">#ifdef OPENSSL_FIPS</div>
<div class=""> int mode =
FIPS_mode();</div>
<div class=""> Debug("ssl",
"FIPS_mode: %d", mode);</div>
<div class=""> mode = FIPS_mode();</div>
<div class=""> if (mode) {</div>
<div class=""> FIPS_mode_set(0);</div>
<div class=""> Debug("ssl",
"FIPS_mode: %d", mode);</div>
<div class=""> }</div>
<div class="">#endif</div>
</div>
</div>
<div class=""><br class="">
</div>
<div class="">
<div class="">[Apr 24 21:43:45.860]
Server {0x7f7628146800} DEBUG: (ssl)
FIPS_mode: 0</div>
<div class=""><br class="">
</div>
<div class="">
<div style="letter-spacing: normal;
orphans: auto; text-align: start;
text-indent: 0px; text-transform:
none; white-space: normal; widows:
auto; word-spacing: 0px;
-webkit-text-stroke-width: 0px;
word-wrap: break-word;
-webkit-nbsp-mode: space;
-webkit-line-break:
after-white-space;" class="">
<div class="">-Bryan</div>
<div class=""><br class="">
</div>
</div>
<br class="Apple-interchange-newline">
<br class="Apple-interchange-newline">
</div>
<br class="">
<div class="">
<blockquote type="cite" class="">
<div class="">On Apr 24, 2015, at
10:56 AM, John Foley <<a
moz-do-not-send="true"
href="mailto:foleyj@cisco.com"
class="">foleyj@cisco.com</a>>
wrote:</div>
<br
class="Apple-interchange-newline">
<div class="">
<div bgcolor="#FFFFFF"
text="#000000" class=""> When
you create the private key given
to SSL_CTX, be sure to
instantiate a private key for
each thread as well. If you
share the same key across all
threads, this can introduce
contention.<br class="">
<br class="">
Here are the type 1 & 2
locks:<br class="">
<br class="">
# define
CRYPTO_LOCK_ERR
1<br class="">
# define
CRYPTO_LOCK_EX_DATA
2<br class="">
<br class="">
Have you checked the OpenSSL
error log to see what errors are
being generated?<br class="">
<br class="">
Also, if you disable FIPS, does
the thread contention still
occur?<br class="">
<br class="">
<br class="">
<br class="">
<div class="moz-cite-prefix">On
04/24/2015 01:36 PM, Bryan
Call wrote:<br class="">
</div>
<blockquote
cite="mid:26886ABC-ABED-4673-847C-5030C5C343D3@apache.org"
type="cite" class=""> We are
using a single SSL_CTX across
all the threads, so I will
create multiple SSL_CTX per
thread. I implemented dynlock
callbacks yesterday, but I
didn’t see them being used in
the tests I did. I also added
atomic counters to see what
type of locks are having
contention (see logs below):
<div class=""><br class="">
</div>
<div class="">Most of the lock
contention seems to be
around type 2 and type 1
locks when calling our lock
callback. In the FIPS/POST
and SSL_CTX recommendations,
in your previous email, what
types of locks would those
help with?</div>
<div class=""><br class="">
</div>
<div class="">Log output:</div>
<div class="">
<div style="margin: 0px;"
class=""><font class=""
color="#931a68"
face="Monaco"><span
style="font-size:
11px;" class="">[Apr
24 10:21:24.339]
Server {0xb310000}
ERROR: <<a
moz-do-not-send="true"
href="http://sslutils.cc/" class="">SSLUtils.cc</a>:178
(SSL_locking_callback)>
contention for lock -
total contention: 3938
waiting: 1 file:
/SourceCache/OpenSSL098/OpenSSL098-52.20.2/src/crypto/ex_data.c
line: 496 type: 2</span></font></div>
</div>
<div style="margin: 0px;"
class=""><span
style="font-size: 11px;
color: rgb(147, 26, 104);
font-family: Monaco;"
class="">[Apr 24
10:21:24.754] Server
{0xb10a000} ERROR: <<a
moz-do-not-send="true"
href="http://sslutils.cc/"
class="">SSLUtils.cc</a>:178
(SSL_locking_callback)>
contention for lock -
total contention: 1350
waiting: 1 file:
/SourceCache/OpenSSL098/OpenSSL098-52.20.2/src/crypto/err/err_def.c
line: 343 type: 1</span></div>
<div style="margin: 0px;"
class=""><br class="">
</div>
<div style="margin: 0px;"
class=""><font class=""
color="#931a68"
face="Monaco"><span
style="font-size: 11px;"
class=""><br class="">
</span></font></div>
<div class="">
<div class="">
<div class="">
<div
style="letter-spacing:
normal; orphans: auto;
text-align: start;
text-indent: 0px;
text-transform: none;
white-space: normal;
widows: auto;
word-spacing: 0px;
-webkit-text-stroke-width:
0px; word-wrap:
break-word;
-webkit-nbsp-mode:
space;
-webkit-line-break:
after-white-space;"
class="">
<div class="">-Bryan</div>
<div class=""><br
class="">
</div>
</div>
<br
class="Apple-interchange-newline">
<br
class="Apple-interchange-newline">
</div>
<br class="">
<div class="">
<blockquote type="cite"
class="">
<div class="">On Apr
23, 2015, at 4:46
PM, John Foley <<a
moz-do-not-send="true" href="mailto:foleyj@cisco.com" class="">foleyj@cisco.com</a>>
wrote:</div>
<br
class="Apple-interchange-newline">
<div class="">
<div
bgcolor="#FFFFFF"
text="#000000"
class=""> Looking
at your call
stack, it appears
you're testing
with FIPS mode
enabled. There's
a lock in
FIPS_selftest_failed()
that's used to
ensure the crypto
algorithms inside
the FIPS boundary
are not utilized
until after the
POST has completed
and passed.
Unfortunately this
lock remains
installed for the
lifetime of the
process. We've
seen thread
contention in this
area when using
OpenSSL. One
work-around is to
modify your
handler that you
pass to
SSL_locking_callback()
to not lock when
the lock ID is
CRYPTO_LOCK_FIPS.
But you would only
want to disable
this particular
lock ID after the
POST has
completed.<br
class="">
<br class="">
If the above
doesn't help,
another item to
check is whether
you're
instantiating an
SSL_CTX for each
thread. I've seen
some developers
use a single
SSL_CTX for
multiple threads.
This results in a
single instance of
the server cert
and private key
being shared
across all the
threads. This
results in thread
contention on the
asymmetric key
operations
associated with
the cert and
private key (e.g.
RSA). Be sure
that each thread
instantiates it's
own SSL_CTX and
certificate/private
key.<br class="">
<br class="">
Lastly, add some
debugs to count
the locks by lock
ID in your lock
handler. There
are about 40
different lock IDs
defined in
crypto/crypto.h.
You'll need to
identify which
lock is causing
the thread
contention. Once
you know that,
then you can look
in the code to see
where that lock is
used and
troubleshoot from
there.<br class="">
<br class="">
<br class="">
<br class="">
<div
class="moz-cite-prefix">On
04/23/2015 07:23
PM, Bryan Call
wrote:<br
class="">
</div>
<blockquote
cite="mid:20A62096-F3DE-4A11-8A76-CE081638E659@apache.org"
type="cite"
class=""> I was
running a
benchmark on a
28 core (56
hyper-threaded)
server that is
running 84
threads in the
process and I am
seeing a lot of
lock contention.
I saw a lot of
lock contention
in the calls to
SSL_get_error()
ssl3_accept().
I am running
RHEL 6.5
and openssl-1.0.1e-30.el6_6.7.x86_64.
We have
used SSL_locking_callback()
to setup pthread
mutex locks, but
haven’t setup
dynlocks yet.
Would dynlocks
help in this
scenario?
<div class=""><br
class="">
</div>
<div class="">
<div class=""><br
class="">
</div>
<div class="">Output
from perf:</div>
<div class="">
<div class="">Samples:
1M of event
'cycles',
Event count
(approx.):
602783506611</div>
<div class="">-
48.04% [ET_NET
0]
[kernel.kallsyms]
[k] _spin_lock
◆</div>
<div class="">-
_spin_lock ▒</div>
<ul class="">
<li class="">+
48.05%
futex_wait_setup
▒</li>
<li class="">+
47.79%
futex_wake ▒</li>
<li class="">+
1.20%
tcp_v4_rcv ▒</li>
<li class="">+
0.98%
task_rq_lock ▒</li>
<li class="">+
0.64%
double_lock_balance
▒</li>
</ul>
<div class="">+
3.94% [ET_NET
0]
libcrypto.so.1.0.1e
[.]
bn_mul_mont ▒</div>
<div class="">+
1.93% [ET_NET
0]
libcrypto.so.1.0.1e
[.] BN_usub ▒</div>
<div class="">+
1.80% [ET_NET
0]
libcrypto.so.1.0.1e
[.]
0x00000000000e5200
▒</div>
<div class="">+
1.49% [ET_NET
0]
libcrypto.so.1.0.1e
[.]
sha256_block_data_order
▒</div>
<div class="">+
0.97% [ET_NET
0]
libcrypto.so.1.0.1e
[.]
gcm_ghash_clmul
▒</div>
<div class="">+
0.85% [ET_NET
0]
libpthread-2.12.so
[.]
pthread_mutex_lock
▒</div>
<div class="">+
0.84% [ET_NET
0]
libcrypto.so.1.0.1e
[.]
aesni_ctr32_encrypt_blocks
▒</div>
<div class="">+
0.74% [ET_NET
0]
libc-2.12.so
[.] malloc ▒</div>
<div class="">+
0.72% [ET_NET
0]
libc-2.12.so
[.] memcpy ▒</div>
<div class="">+
0.57% [ET_NET
0]
libcrypto.so.1.0.1e
[.]
BN_mod_mul_montgomery
▒</div>
<div class="">+
0.51% [ET_NET
0]
libpthread-2.12.so
[.]
pthread_mutex_unlock</div>
<div class=""><br
class="webkit-block-placeholder">
</div>
<div class="">pstack
of the process
(only a couple
of the
interesting
threads):</div>
<div class="">
<div class=""><span
style="font-family:
'Andale
Mono';"
class="">Thread
1 (Thread
0x7f1fba151800
(LWP 1106)):</span></div>
<div class=""><span
style="font-family:
'Andale
Mono';"
class="">#0
0x00007f1fbaaeb264
in
__lll_lock_wait
() from
/lib64/libpthread.so.0</span></div>
<div class=""><span
style="font-family:
'Andale
Mono';"
class="">#1
0x00007f1fbaae6508
in _L_lock_854
() from
/lib64/libpthread.so.0</span></div>
<div class=""><span
style="font-family:
'Andale
Mono';"
class="">#2
0x00007f1fbaae63d7
in
pthread_mutex_lock
() from
/lib64/libpthread.so.0</span></div>
<div class=""><span
style="font-family:
'Andale
Mono';"
class="">#3
0x000000000072822f
in
SSL_locking_callback(int,
int, char
const*, int)
()</span></div>
<div class=""><span
style="font-family:
'Andale
Mono';"
class="">#4
0x00000031d2ae7f03
in ?? () from
/usr/lib64/libcrypto.so.10</span></div>
<div class=""><span
style="font-family:
'Andale
Mono';"
class="">#5
0x00000031d2ae7d8c
in ?? () from
/usr/lib64/libcrypto.so.10</span></div>
<div class=""><span
style="font-family:
'Andale
Mono';"
class="">#6
0x00000031d2ae80ab
in
ERR_get_state
() from
/usr/lib64/libcrypto.so.10</span></div>
<div class=""><span
style="font-family:
'Andale
Mono';"
class="">#7
0x00000031d2ae8557
in ?? () from
/usr/lib64/libcrypto.so.10</span></div>
<div class=""><span
style="font-family:
'Andale
Mono';"
class="">#8
0x00000031d2e424a9
in
SSL_get_error
() from
/usr/lib64/libssl.so.10</span></div>
<div class=""><span
style="font-family:
'Andale
Mono';"
class="">#9
0x00000000007249ef
in
SSLNetVConnection::load_buffer_and_write(long,
long&,
long&,
MIOBufferAccessor&,
int&) ()</span></div>
<div class=""><span
style="font-family:
'Andale
Mono';"
class="">#10
0x00000000007389df
in
write_to_net_io(NetHandler*,
UnixNetVConnection*,
EThread*) ()</span></div>
<div class=""><span
style="font-family:
'Andale
Mono';"
class="">#11
0x0000000000738523
in
write_to_net(NetHandler*,
UnixNetVConnection*,
EThread*) ()</span></div>
<div class=""><span
style="font-family:
'Andale
Mono';"
class="">#12
0x0000000000731922
in
NetHandler::mainNetEvent(int,
Event*) ()</span></div>
<div class=""><span
style="font-family:
'Andale
Mono';"
class="">#13
0x00000000004f6d48
in
Continuation::handleEvent(int,
void*) ()</span></div>
<div class=""><span
style="font-family:
'Andale
Mono';"
class="">#14
0x000000000075877a
in
EThread::process_event(Event*,
int) ()</span></div>
<div class=""><span
style="font-family:
'Andale
Mono';"
class="">#15
0x0000000000758c84
in EThread::ex</span></div>
<div class=""><span
style="font-family:
'Andale
Mono';"
class=""><br
class="">
</span></div>
<div class=""><span
style="font-family:
'Andale
Mono';"
class="">Thread
56 (Thread
0x7f1fb0ecd700
(LWP 1172)):</span></div>
<div class=""><span
style="font-family:
'Andale
Mono';"
class="">#0
0x00007f1fbaae77d9
in
pthread_mutex_unlock
() from
/lib64/libpthread.so.0</span></div>
<div class=""><span
style="font-family:
'Andale
Mono';"
class="">#1
0x0000000000728265
in
SSL_locking_callback(int,
int, char
const*, int)
()</span></div>
<div class=""><span
style="font-family:
'Andale
Mono';"
class="">#2
0x00000031d2b57402
in ?? () from
/usr/lib64/libcrypto.so.10</span></div>
<div class=""><span
style="font-family:
'Andale
Mono';"
class="">#3
0x00000031d2b574bd
in
FIPS_selftest_failed
() from
/usr/lib64/libcrypto.so.10</span></div>
<div class=""><span
style="font-family:
'Andale
Mono';"
class="">#4
0x00000031d2ae9f1c
in
EVP_DigestInit_ex
() from
/usr/lib64/libcrypto.so.10</span></div>
<div class=""><span
style="font-family:
'Andale
Mono';"
class="">#5
0x00000031d2a7977e
in
HMAC_Init_ex
() from
/usr/lib64/libcrypto.so.10</span></div>
<div class=""><span
style="font-family:
'Andale
Mono';"
class="">#6
0x00000031d2a79c29
in ?? () from
/usr/lib64/libcrypto.so.10</span></div>
<div class=""><span
style="font-family:
'Andale
Mono';"
class="">#7
0x00000031d2af7a42
in
EVP_PKEY_CTX_ctrl
() from
/usr/lib64/libcrypto.so.10</span></div>
<div class=""><span
style="font-family:
'Andale
Mono';"
class="">#8
0x00000031d2ae9ffb
in
EVP_DigestInit_ex
() from
/usr/lib64/libcrypto.so.10</span></div>
<div class=""><span
style="font-family:
'Andale
Mono';"
class="">#9
0x00000031d2af952a
in ?? () from
/usr/lib64/libcrypto.so.10</span></div>
<div class=""><span
style="font-family:
'Andale
Mono';"
class="">#10
0x00000031d2e34cf2
in tls1_PRF ()
from
/usr/lib64/libssl.so.10</span></div>
<div class=""><span
style="font-family:
'Andale
Mono';"
class="">#11
0x00000031d2e35796
in
tls1_final_finish_mac
() from
/usr/lib64/libssl.so.10</span></div>
<div class=""><span
style="font-family:
'Andale
Mono';"
class="">#12
0x00000031d2e2db2e
in
ssl3_get_message
() from
/usr/lib64/libssl.so.10</span></div>
<div class=""><span
style="font-family:
'Andale
Mono';"
class="">#13
0x00000031d2e1dc3c
in
ssl3_get_cert_verify
() from
/usr/lib64/libssl.so.10</span></div>
<div class=""><span
style="font-family:
'Andale
Mono';"
class="">#14
0x00000031d2e220d6
in ssl3_accept
() from
/usr/lib64/libssl.so.10</span></div>
<div class=""><span
style="font-family:
'Andale
Mono';"
class="">#15
0x0000000000725287
in
SSLNetVConnection::sslServerHandShakeEvent(int&)
()</span></div>
<div class=""><span
style="font-family:
'Andale
Mono';"
class="">#16
0x00000000007251b3
in
SSLNetVConnection::sslStartHandShake(int,
int&) ()</span></div>
<div class=""><span
style="font-family:
'Andale
Mono';"
class="">#17
0x0000000000723df3
in
SSLNetVConnection::net_read_io(NetHandler*,
EThread*) ()</span></div>
<div class=""><span
style="font-family:
'Andale
Mono';"
class="">#18
0x0000000000731852
in
NetHandler::mainNetEvent(int,
Event*) ()</span></div>
<div class=""><span
style="font-family:
'Andale
Mono';"
class="">#19
0x00000000004f6d48
in
Continuation::handleEvent(int,
void*) ()</span></div>
<div class=""><span
style="font-family:
'Andale
Mono';"
class="">#20
0x000000000075877a
in
EThread::process_event(Event*,
int) ()</span></div>
<div class=""><span
style="font-family:
'Andale
Mono';"
class="">#21
0x0000000000758c84
in
EThread::execute()
()</span></div>
<div class=""><span
style="font-family:
'Andale
Mono';"
class="">#22
0x0000000000757cf8
in
spawn_thread_internal(void*)
()</span></div>
<div class=""><span
style="font-family:
'Andale
Mono';"
class="">#23
0x00007f1fbaae49d1
in
start_thread
() from
/lib64/libpthread.so.0</span></div>
<div class=""><span
style="font-family:
'Andale
Mono';"
class="">#24
0x00000030ff0e88fd
in clone ()
from
/lib64/libc.so.6</span></div>
</div>
<div class=""><br
class="">
</div>
<div class="">
<div
style="letter-spacing:
normal;
orphans: auto;
text-align:
start;
text-indent:
0px;
text-transform:
none;
white-space:
normal;
widows: auto;
word-spacing:
0px;
-webkit-text-stroke-width:
0px;
word-wrap:
break-word;
-webkit-nbsp-mode:
space;
-webkit-line-break:
after-white-space;" class="">
<div class="">-Bryan</div>
<div class=""><br
class="">
</div>
</div>
<br
class="Apple-interchange-newline">
<br
class="Apple-interchange-newline">
</div>
<br class="">
</div>
</div>
<br class="">
<fieldset
class="mimeAttachmentHeader"></fieldset>
<br class="">
<pre class="" wrap="">_______________________________________________
openssl-users mailing list
To unsubscribe: <a moz-do-not-send="true" class="moz-txt-link-freetext" href="https://mta.openssl.org/mailman/listinfo/openssl-users">https://mta.openssl.org/mailman/listinfo/openssl-users</a>
</pre>
</blockquote>
<br class="">
</div>
_______________________________________________<br class="">
openssl-users
mailing list<br
class="">
To unsubscribe: <a
moz-do-not-send="true"
href="https://mta.openssl.org/mailman/listinfo/openssl-users"
class="">https://mta.openssl.org/mailman/listinfo/openssl-users</a><br
class="">
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
<br class="">
<fieldset
class="mimeAttachmentHeader"></fieldset>
<br class="">
<pre class="" wrap="">_______________________________________________
openssl-users mailing list
To unsubscribe: <a moz-do-not-send="true" class="moz-txt-link-freetext" href="https://mta.openssl.org/mailman/listinfo/openssl-users">https://mta.openssl.org/mailman/listinfo/openssl-users</a>
</pre>
</blockquote>
<br class="">
</div>
_______________________________________________<br class="">
openssl-users mailing list<br
class="">
To unsubscribe: <a
moz-do-not-send="true"
href="https://mta.openssl.org/mailman/listinfo/openssl-users"
class="">https://mta.openssl.org/mailman/listinfo/openssl-users</a><br
class="">
</div>
</blockquote>
</div>
<br class="">
</div>
<br class="">
<fieldset class="mimeAttachmentHeader"></fieldset>
<br class="">
<pre class="" wrap="">_______________________________________________
openssl-users mailing list
To unsubscribe: <a moz-do-not-send="true" class="moz-txt-link-freetext" href="https://mta.openssl.org/mailman/listinfo/openssl-users">https://mta.openssl.org/mailman/listinfo/openssl-users</a>
</pre>
</blockquote>
<br class="">
</div>
_______________________________________________<br class="">
openssl-users mailing list<br class="">
To unsubscribe: <a moz-do-not-send="true"
href="https://mta.openssl.org/mailman/listinfo/openssl-users"
class="">https://mta.openssl.org/mailman/listinfo/openssl-users</a><br
class="">
</div>
</blockquote>
</div>
<br class="">
</div>
<br class="">
<fieldset class="mimeAttachmentHeader"></fieldset>
<br class="">
<pre class="" wrap="">_______________________________________________
openssl-users mailing list
To unsubscribe: <a moz-do-not-send="true" class="moz-txt-link-freetext" href="https://mta.openssl.org/mailman/listinfo/openssl-users">https://mta.openssl.org/mailman/listinfo/openssl-users</a>
</pre>
</blockquote>
<br class="">
</div>
_______________________________________________<br
class="">
openssl-users mailing list<br class="">
To unsubscribe: <a moz-do-not-send="true"
href="https://mta.openssl.org/mailman/listinfo/openssl-users"
class="">https://mta.openssl.org/mailman/listinfo/openssl-users</a><br
class="">
</div>
</blockquote>
</div>
<br class="">
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
openssl-users mailing list
To unsubscribe: <a class="moz-txt-link-freetext" href="https://mta.openssl.org/mailman/listinfo/openssl-users">https://mta.openssl.org/mailman/listinfo/openssl-users</a>
</pre>
</blockquote>
<br>
</body>
</html>