<html>
<head>
<meta content="text/html; charset=UTF-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
In the context of OpenSSL 1.0.1 or 1.0.2, this means
FIPS_mode_set(1) has returned with a successful return value. <br>
<br>
In the future, this may change if/when OpenSSL does another FIPS
validation. CMVP has changed the implementation guidance that
requires the POST to run automatically without user intervention.
Any future FIPS validations would be subject to the new rules.
Hence, the behavior of FIPS_mode_set() may change in the future if
OpenSSL decides to pursue another validation.<br>
<br>
<br>
<br>
<div class="moz-cite-prefix">On 04/28/2015 12:43 PM, Bryan Call
wrote:<br>
</div>
<blockquote
cite="mid:08D500E9-5A78-4CB8-B49B-6F41523A2C2E@apache.org"
type="cite">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
What do you mean by “FIPS POST has completed”?
<div class=""><br class="">
<div class="">
<div style="color: rgb(0, 0, 0); letter-spacing: normal;
orphans: auto; text-align: start; text-indent: 0px;
text-transform: none; white-space: normal; widows: auto;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
word-wrap: break-word; -webkit-nbsp-mode: space;
-webkit-line-break: after-white-space;" class="">
<div class="">-Bryan</div>
<div class=""><br class="">
</div>
</div>
<br class="Apple-interchange-newline">
<br class="Apple-interchange-newline">
</div>
<br class="">
<div>
<blockquote type="cite" class="">
<div class="">On Apr 24, 2015, at 4:17 PM, John Foley <<a
moz-do-not-send="true" href="mailto:foleyj@cisco.com"
class="">foleyj@cisco.com</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div bgcolor="#FFFFFF" text="#000000" class=""> Some of
the algorithms still invoke the FIPS flavor of the
algorithm even when FIPS is disabled. For example, this
code is from EVP_DigestUpdate().<br class="">
<br class="">
int EVP_DigestUpdate(EVP_MD_CTX *ctx, const void *data,
size_t count)<br class="">
{<br class="">
#ifdef OPENSSL_FIPS<br class="">
return FIPS_digestupdate(ctx, data, count);<br
class="">
#else<br class="">
return ctx->update(ctx, data, count);<br class="">
#endif<br class="">
}<br class="">
<br class="">
In 1.0.1, locks 39 and 40 are:<br class="">
<br class="">
# define CRYPTO_LOCK_FIPS 39<br class="">
# define CRYPTO_LOCK_FIPS2 40<br class="">
<br class="">
You should try adding some logic to skip the lock in
your lock handler when the lock ID is 39 or 40. Again,
it should be safe to not lock on these two as long as
the FIPS POST has completed.<br class="">
<br class="">
<br class="">
<br class="">
<div class="moz-cite-prefix">On 04/24/2015 05:56 PM,
Bryan Call wrote:<br class="">
</div>
<blockquote
cite="mid:A5F20DF3-E379-4952-82BB-C3CB3A4BC5DA@apache.org"
type="cite" class=""> In my last email I ran the
benchmark on Fedora 21 (big mistake). Here are the
results when running it back on the 28/56 core RHEL
6.5 server showing contention in a different place
(fips). Is there a reason it would be calling into
the FIPS code to get a lock of FIPS is not enabled?
<div class="">
<div class=""><br class="">
</div>
</div>
<div class="">Contention log:</div>
<div class="">
<div class="">[Apr 24 21:35:09.731] Server
{0x7f5529ccd700} ERROR: contention for lock -
total contention: 70662 waiting: 16 file: fips.c
line: 471 type: 40</div>
<div class="">[Apr 24 21:35:09.732] Server
{0x7f55299ca700} ERROR: contention for lock -
total contention: 71605 waiting: 2 file: fips.c
line: 107 type: 39</div>
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class="">Line in in fips that see the contention
the most:</div>
<div class="">
<div class="">[bcall@bm77 trafficserver]$ grep
fips.c diags.log | cut -f 19 -d ' ' | sort | uniq
-c | sort -rn</div>
<div class=""> 875379 471</div>
<div class=""> 288338 107</div>
<div class=""> 275472 127</div>
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class="">Different lock types and their
contention frequency (multiple benchmarks):</div>
<div class="">
<div class=""> 875379 40 < - in fips.c</div>
<div class=""> 563810 39 <- also in fips.c</div>
<div class=""> 440518 1</div>
<div class=""> 44145 18</div>
<div class=""> 17447 2</div>
<div class=""> 11356 10</div>
<div class=""> 1377 19</div>
<div class=""> 687 12</div>
<div class=""> 532 25</div>
<div class=""> 302 33</div>
<div class=""> 239 9</div>
<div class=""> 58 16</div>
<div class=""> 40 21</div>
<div class=""> 32 11</div>
<div class=""> 19 3</div>
<div class=""> 1 14</div>
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class="">Code to diable FIPS and log:</div>
<div class="">
<div class="">
<div class="">#ifdef OPENSSL_FIPS</div>
<div class=""> int mode = FIPS_mode();</div>
<div class=""> Debug("ssl", "FIPS_mode: %d",
mode);</div>
<div class=""> mode = FIPS_mode();</div>
<div class=""> if (mode) {</div>
<div class=""> FIPS_mode_set(0);</div>
<div class=""> Debug("ssl", "FIPS_mode: %d",
mode);</div>
<div class=""> }</div>
<div class="">#endif</div>
</div>
</div>
<div class=""><br class="">
</div>
<div class="">
<div class="">[Apr 24 21:43:45.860] Server
{0x7f7628146800} DEBUG: (ssl) FIPS_mode: 0</div>
<div class=""><br class="">
</div>
<div class="">
<div style="letter-spacing: normal; orphans: auto;
text-align: start; text-indent: 0px;
text-transform: none; white-space: normal;
widows: auto; word-spacing: 0px;
-webkit-text-stroke-width: 0px; word-wrap:
break-word; -webkit-nbsp-mode: space;
-webkit-line-break: after-white-space;" class="">
<div class="">-Bryan</div>
<div class=""><br class="">
</div>
</div>
<br class="Apple-interchange-newline">
<br class="Apple-interchange-newline">
</div>
<br class="">
<div class="">
<blockquote type="cite" class="">
<div class="">On Apr 24, 2015, at 10:56 AM, John
Foley <<a moz-do-not-send="true"
href="mailto:foleyj@cisco.com" class="">foleyj@cisco.com</a>>
wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div bgcolor="#FFFFFF" text="#000000" class="">
When you create the private key given to
SSL_CTX, be sure to instantiate a private
key for each thread as well. If you share
the same key across all threads, this can
introduce contention.<br class="">
<br class="">
Here are the type 1 & 2 locks:<br
class="">
<br class="">
# define CRYPTO_LOCK_ERR 1<br
class="">
# define CRYPTO_LOCK_EX_DATA 2<br
class="">
<br class="">
Have you checked the OpenSSL error log to
see what errors are being generated?<br
class="">
<br class="">
Also, if you disable FIPS, does the thread
contention still occur?<br class="">
<br class="">
<br class="">
<br class="">
<div class="moz-cite-prefix">On 04/24/2015
01:36 PM, Bryan Call wrote:<br class="">
</div>
<blockquote
cite="mid:26886ABC-ABED-4673-847C-5030C5C343D3@apache.org"
type="cite" class=""> We are using a
single SSL_CTX across all the threads, so
I will create multiple SSL_CTX per thread.
I implemented dynlock callbacks
yesterday, but I didn’t see them being
used in the tests I did. I also added
atomic counters to see what type of locks
are having contention (see logs below):
<div class=""><br class="">
</div>
<div class="">Most of the lock contention
seems to be around type 2 and type 1
locks when calling our lock callback.
In the FIPS/POST and SSL_CTX
recommendations, in your previous email,
what types of locks would those help
with?</div>
<div class=""><br class="">
</div>
<div class="">Log output:</div>
<div class="">
<div style="margin: 0px;" class=""><font
class="" color="#931a68"
face="Monaco"><span
style="font-size: 11px;" class="">[Apr
24 10:21:24.339] Server
{0xb310000} ERROR: <<a
moz-do-not-send="true"
href="http://sslutils.cc/"
class="">SSLUtils.cc</a>:178
(SSL_locking_callback)>
contention for lock - total
contention: 3938 waiting: 1 file:
/SourceCache/OpenSSL098/OpenSSL098-52.20.2/src/crypto/ex_data.c
line: 496 type: 2</span></font></div>
</div>
<div style="margin: 0px;" class=""><span
style="font-size: 11px; color:
rgb(147, 26, 104); font-family:
Monaco;" class="">[Apr 24
10:21:24.754] Server {0xb10a000}
ERROR: <<a moz-do-not-send="true"
href="http://sslutils.cc/" class="">SSLUtils.cc</a>:178
(SSL_locking_callback)> contention
for lock - total contention: 1350
waiting: 1 file:
/SourceCache/OpenSSL098/OpenSSL098-52.20.2/src/crypto/err/err_def.c
line: 343 type: 1</span></div>
<div style="margin: 0px;" class=""><br
class="">
</div>
<div style="margin: 0px;" class=""><font
class="" color="#931a68" face="Monaco"><span
style="font-size: 11px;" class=""><br
class="">
</span></font></div>
<div class="">
<div class="">
<div class="">
<div style="letter-spacing: normal;
orphans: auto; text-align: start;
text-indent: 0px; text-transform:
none; white-space: normal; widows:
auto; word-spacing: 0px;
-webkit-text-stroke-width: 0px;
word-wrap: break-word;
-webkit-nbsp-mode: space;
-webkit-line-break:
after-white-space;" class="">
<div class="">-Bryan</div>
<div class=""><br class="">
</div>
</div>
<br
class="Apple-interchange-newline">
<br
class="Apple-interchange-newline">
</div>
<br class="">
<div class="">
<blockquote type="cite" class="">
<div class="">On Apr 23, 2015, at
4:46 PM, John Foley <<a
moz-do-not-send="true"
href="mailto:foleyj@cisco.com"
class="">foleyj@cisco.com</a>>
wrote:</div>
<br
class="Apple-interchange-newline">
<div class="">
<div bgcolor="#FFFFFF"
text="#000000" class="">
Looking at your call stack, it
appears you're testing with
FIPS mode enabled. There's a
lock in FIPS_selftest_failed()
that's used to ensure the
crypto algorithms inside the
FIPS boundary are not utilized
until after the POST has
completed and passed.
Unfortunately this lock
remains installed for the
lifetime of the process.
We've seen thread contention
in this area when using
OpenSSL. One work-around is
to modify your handler that
you pass to
SSL_locking_callback() to not
lock when the lock ID is
CRYPTO_LOCK_FIPS. But you
would only want to disable
this particular lock ID after
the POST has completed.<br
class="">
<br class="">
If the above doesn't help,
another item to check is
whether you're instantiating
an SSL_CTX for each thread.
I've seen some developers use
a single SSL_CTX for multiple
threads. This results in a
single instance of the server
cert and private key being
shared across all the
threads. This results in
thread contention on the
asymmetric key operations
associated with the cert and
private key (e.g. RSA). Be
sure that each thread
instantiates it's own SSL_CTX
and certificate/private key.<br
class="">
<br class="">
Lastly, add some debugs to
count the locks by lock ID in
your lock handler. There are
about 40 different lock IDs
defined in crypto/crypto.h.
You'll need to identify which
lock is causing the thread
contention. Once you know
that, then you can look in the
code to see where that lock is
used and troubleshoot from
there.<br class="">
<br class="">
<br class="">
<br class="">
<div class="moz-cite-prefix">On
04/23/2015 07:23 PM, Bryan
Call wrote:<br class="">
</div>
<blockquote
cite="mid:20A62096-F3DE-4A11-8A76-CE081638E659@apache.org"
type="cite" class=""> I was
running a benchmark on a 28
core (56 hyper-threaded)
server that is running 84
threads in the process and I
am seeing a lot of lock
contention. I saw a lot of
lock contention in the calls
to SSL_get_error()
ssl3_accept(). I am running
RHEL 6.5
and openssl-1.0.1e-30.el6_6.7.x86_64.
We have
used SSL_locking_callback()
to setup pthread mutex
locks, but haven’t setup
dynlocks yet. Would
dynlocks help in this
scenario?
<div class=""><br class="">
</div>
<div class="">
<div class=""><br class="">
</div>
<div class="">Output from
perf:</div>
<div class="">
<div class="">Samples:
1M of event 'cycles',
Event count (approx.):
602783506611</div>
<div class="">- 48.04%
[ET_NET 0]
[kernel.kallsyms] [k]
_spin_lock ◆</div>
<div class="">-
_spin_lock ▒</div>
<ul class="">
<li class="">+ 48.05%
futex_wait_setup ▒</li>
<li class="">+ 47.79%
futex_wake ▒</li>
<li class="">+ 1.20%
tcp_v4_rcv ▒</li>
<li class="">+ 0.98%
task_rq_lock ▒</li>
<li class="">+ 0.64%
double_lock_balance
▒</li>
</ul>
<div class="">+ 3.94%
[ET_NET 0]
libcrypto.so.1.0.1e
[.] bn_mul_mont ▒</div>
<div class="">+ 1.93%
[ET_NET 0]
libcrypto.so.1.0.1e
[.] BN_usub ▒</div>
<div class="">+ 1.80%
[ET_NET 0]
libcrypto.so.1.0.1e
[.] 0x00000000000e5200
▒</div>
<div class="">+ 1.49%
[ET_NET 0]
libcrypto.so.1.0.1e
[.]
sha256_block_data_order
▒</div>
<div class="">+ 0.97%
[ET_NET 0]
libcrypto.so.1.0.1e
[.] gcm_ghash_clmul ▒</div>
<div class="">+ 0.85%
[ET_NET 0]
libpthread-2.12.so [.]
pthread_mutex_lock ▒</div>
<div class="">+ 0.84%
[ET_NET 0]
libcrypto.so.1.0.1e
[.]
aesni_ctr32_encrypt_blocks
▒</div>
<div class="">+ 0.74%
[ET_NET 0]
libc-2.12.so [.]
malloc ▒</div>
<div class="">+ 0.72%
[ET_NET 0]
libc-2.12.so [.]
memcpy ▒</div>
<div class="">+ 0.57%
[ET_NET 0]
libcrypto.so.1.0.1e
[.]
BN_mod_mul_montgomery
▒</div>
<div class="">+ 0.51%
[ET_NET 0]
libpthread-2.12.so [.]
pthread_mutex_unlock</div>
<div class=""><br
class="webkit-block-placeholder">
</div>
<div class="">pstack of
the process (only a
couple of the
interesting threads):</div>
<div class="">
<div class=""><span
style="font-family:
'Andale Mono';"
class="">Thread 1
(Thread
0x7f1fba151800
(LWP 1106)):</span></div>
<div class=""><span
style="font-family:
'Andale Mono';"
class="">#0
0x00007f1fbaaeb264
in __lll_lock_wait
() from
/lib64/libpthread.so.0</span></div>
<div class=""><span
style="font-family:
'Andale Mono';"
class="">#1
0x00007f1fbaae6508
in _L_lock_854 ()
from
/lib64/libpthread.so.0</span></div>
<div class=""><span
style="font-family:
'Andale Mono';"
class="">#2
0x00007f1fbaae63d7
in
pthread_mutex_lock
() from
/lib64/libpthread.so.0</span></div>
<div class=""><span
style="font-family:
'Andale Mono';"
class="">#3
0x000000000072822f
in
SSL_locking_callback(int,
int, char const*,
int) ()</span></div>
<div class=""><span
style="font-family:
'Andale Mono';"
class="">#4
0x00000031d2ae7f03
in ?? () from
/usr/lib64/libcrypto.so.10</span></div>
<div class=""><span
style="font-family:
'Andale Mono';"
class="">#5
0x00000031d2ae7d8c
in ?? () from
/usr/lib64/libcrypto.so.10</span></div>
<div class=""><span
style="font-family:
'Andale Mono';"
class="">#6
0x00000031d2ae80ab
in ERR_get_state
() from
/usr/lib64/libcrypto.so.10</span></div>
<div class=""><span
style="font-family:
'Andale Mono';"
class="">#7
0x00000031d2ae8557
in ?? () from
/usr/lib64/libcrypto.so.10</span></div>
<div class=""><span
style="font-family:
'Andale Mono';"
class="">#8
0x00000031d2e424a9
in SSL_get_error
() from
/usr/lib64/libssl.so.10</span></div>
<div class=""><span
style="font-family:
'Andale Mono';"
class="">#9
0x00000000007249ef
in
SSLNetVConnection::load_buffer_and_write(long,
long&,
long&,
MIOBufferAccessor&,
int&) ()</span></div>
<div class=""><span
style="font-family:
'Andale Mono';"
class="">#10
0x00000000007389df
in
write_to_net_io(NetHandler*,
UnixNetVConnection*,
EThread*) ()</span></div>
<div class=""><span
style="font-family:
'Andale Mono';"
class="">#11
0x0000000000738523
in
write_to_net(NetHandler*,
UnixNetVConnection*,
EThread*) ()</span></div>
<div class=""><span
style="font-family:
'Andale Mono';"
class="">#12
0x0000000000731922
in
NetHandler::mainNetEvent(int,
Event*) ()</span></div>
<div class=""><span
style="font-family:
'Andale Mono';"
class="">#13
0x00000000004f6d48
in
Continuation::handleEvent(int,
void*) ()</span></div>
<div class=""><span
style="font-family:
'Andale Mono';"
class="">#14
0x000000000075877a
in
EThread::process_event(Event*,
int) ()</span></div>
<div class=""><span
style="font-family:
'Andale Mono';"
class="">#15
0x0000000000758c84
in EThread::ex</span></div>
<div class=""><span
style="font-family:
'Andale Mono';"
class=""><br
class="">
</span></div>
<div class=""><span
style="font-family:
'Andale Mono';"
class="">Thread 56
(Thread
0x7f1fb0ecd700
(LWP 1172)):</span></div>
<div class=""><span
style="font-family:
'Andale Mono';"
class="">#0
0x00007f1fbaae77d9
in
pthread_mutex_unlock
() from
/lib64/libpthread.so.0</span></div>
<div class=""><span
style="font-family:
'Andale Mono';"
class="">#1
0x0000000000728265
in
SSL_locking_callback(int,
int, char const*,
int) ()</span></div>
<div class=""><span
style="font-family:
'Andale Mono';"
class="">#2
0x00000031d2b57402
in ?? () from
/usr/lib64/libcrypto.so.10</span></div>
<div class=""><span
style="font-family:
'Andale Mono';"
class="">#3
0x00000031d2b574bd
in
FIPS_selftest_failed
() from
/usr/lib64/libcrypto.so.10</span></div>
<div class=""><span
style="font-family:
'Andale Mono';"
class="">#4
0x00000031d2ae9f1c
in
EVP_DigestInit_ex
() from
/usr/lib64/libcrypto.so.10</span></div>
<div class=""><span
style="font-family:
'Andale Mono';"
class="">#5
0x00000031d2a7977e
in HMAC_Init_ex ()
from
/usr/lib64/libcrypto.so.10</span></div>
<div class=""><span
style="font-family:
'Andale Mono';"
class="">#6
0x00000031d2a79c29
in ?? () from
/usr/lib64/libcrypto.so.10</span></div>
<div class=""><span
style="font-family:
'Andale Mono';"
class="">#7
0x00000031d2af7a42
in
EVP_PKEY_CTX_ctrl
() from
/usr/lib64/libcrypto.so.10</span></div>
<div class=""><span
style="font-family:
'Andale Mono';"
class="">#8
0x00000031d2ae9ffb
in
EVP_DigestInit_ex
() from
/usr/lib64/libcrypto.so.10</span></div>
<div class=""><span
style="font-family:
'Andale Mono';"
class="">#9
0x00000031d2af952a
in ?? () from
/usr/lib64/libcrypto.so.10</span></div>
<div class=""><span
style="font-family:
'Andale Mono';"
class="">#10
0x00000031d2e34cf2
in tls1_PRF ()
from
/usr/lib64/libssl.so.10</span></div>
<div class=""><span
style="font-family:
'Andale Mono';"
class="">#11
0x00000031d2e35796
in
tls1_final_finish_mac
() from
/usr/lib64/libssl.so.10</span></div>
<div class=""><span
style="font-family:
'Andale Mono';"
class="">#12
0x00000031d2e2db2e
in
ssl3_get_message
() from
/usr/lib64/libssl.so.10</span></div>
<div class=""><span
style="font-family:
'Andale Mono';"
class="">#13
0x00000031d2e1dc3c
in
ssl3_get_cert_verify
() from
/usr/lib64/libssl.so.10</span></div>
<div class=""><span
style="font-family:
'Andale Mono';"
class="">#14
0x00000031d2e220d6
in ssl3_accept ()
from
/usr/lib64/libssl.so.10</span></div>
<div class=""><span
style="font-family:
'Andale Mono';"
class="">#15
0x0000000000725287
in
SSLNetVConnection::sslServerHandShakeEvent(int&)
()</span></div>
<div class=""><span
style="font-family:
'Andale Mono';"
class="">#16
0x00000000007251b3
in
SSLNetVConnection::sslStartHandShake(int,
int&) ()</span></div>
<div class=""><span
style="font-family:
'Andale Mono';"
class="">#17
0x0000000000723df3
in
SSLNetVConnection::net_read_io(NetHandler*,
EThread*) ()</span></div>
<div class=""><span
style="font-family:
'Andale Mono';"
class="">#18
0x0000000000731852
in
NetHandler::mainNetEvent(int,
Event*) ()</span></div>
<div class=""><span
style="font-family:
'Andale Mono';"
class="">#19
0x00000000004f6d48
in
Continuation::handleEvent(int,
void*) ()</span></div>
<div class=""><span
style="font-family:
'Andale Mono';"
class="">#20
0x000000000075877a
in
EThread::process_event(Event*,
int) ()</span></div>
<div class=""><span
style="font-family:
'Andale Mono';"
class="">#21
0x0000000000758c84
in
EThread::execute()
()</span></div>
<div class=""><span
style="font-family:
'Andale Mono';"
class="">#22
0x0000000000757cf8
in
spawn_thread_internal(void*)
()</span></div>
<div class=""><span
style="font-family:
'Andale Mono';"
class="">#23
0x00007f1fbaae49d1
in start_thread ()
from
/lib64/libpthread.so.0</span></div>
<div class=""><span
style="font-family:
'Andale Mono';"
class="">#24
0x00000030ff0e88fd
in clone () from
/lib64/libc.so.6</span></div>
</div>
<div class=""><br
class="">
</div>
<div class="">
<div
style="letter-spacing:
normal; orphans:
auto; text-align:
start; text-indent:
0px; text-transform:
none; white-space:
normal; widows:
auto; word-spacing:
0px;
-webkit-text-stroke-width:
0px; word-wrap:
break-word;
-webkit-nbsp-mode:
space;
-webkit-line-break:
after-white-space;"
class="">
<div class="">-Bryan</div>
<div class=""><br
class="">
</div>
</div>
<br
class="Apple-interchange-newline">
<br
class="Apple-interchange-newline">
</div>
<br class="">
</div>
</div>
<br class="">
<fieldset
class="mimeAttachmentHeader"></fieldset>
<br class="">
<pre class="" wrap="">_______________________________________________
openssl-users mailing list
To unsubscribe: <a moz-do-not-send="true" class="moz-txt-link-freetext" href="https://mta.openssl.org/mailman/listinfo/openssl-users">https://mta.openssl.org/mailman/listinfo/openssl-users</a>
</pre>
</blockquote>
<br class="">
</div>
_______________________________________________<br class="">
openssl-users mailing list<br
class="">
To unsubscribe: <a
moz-do-not-send="true"
href="https://mta.openssl.org/mailman/listinfo/openssl-users"
class="">https://mta.openssl.org/mailman/listinfo/openssl-users</a><br
class="">
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
<br class="">
<fieldset class="mimeAttachmentHeader"></fieldset>
<br class="">
<pre class="" wrap="">_______________________________________________
openssl-users mailing list
To unsubscribe: <a moz-do-not-send="true" class="moz-txt-link-freetext" href="https://mta.openssl.org/mailman/listinfo/openssl-users">https://mta.openssl.org/mailman/listinfo/openssl-users</a>
</pre>
</blockquote>
<br class="">
</div>
_______________________________________________<br class="">
openssl-users mailing list<br class="">
To unsubscribe: <a moz-do-not-send="true"
href="https://mta.openssl.org/mailman/listinfo/openssl-users"
class="">https://mta.openssl.org/mailman/listinfo/openssl-users</a><br
class="">
</div>
</blockquote>
</div>
<br class="">
</div>
<br class="">
<fieldset class="mimeAttachmentHeader"></fieldset>
<br class="">
<pre class="" wrap="">_______________________________________________
openssl-users mailing list
To unsubscribe: <a moz-do-not-send="true" class="moz-txt-link-freetext" href="https://mta.openssl.org/mailman/listinfo/openssl-users">https://mta.openssl.org/mailman/listinfo/openssl-users</a>
</pre>
</blockquote>
<br class="">
</div>
_______________________________________________<br
class="">
openssl-users mailing list<br class="">
To unsubscribe: <a moz-do-not-send="true"
href="https://mta.openssl.org/mailman/listinfo/openssl-users"
class="">https://mta.openssl.org/mailman/listinfo/openssl-users</a><br
class="">
</div>
</blockquote>
</div>
<br class="">
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
openssl-users mailing list
To unsubscribe: <a class="moz-txt-link-freetext" href="https://mta.openssl.org/mailman/listinfo/openssl-users">https://mta.openssl.org/mailman/listinfo/openssl-users</a>
</pre>
</blockquote>
<br>
</body>
</html>