<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">I was running a benchmark on a 28 core (56 hyper-threaded) server that is running 84 threads in the process and I am seeing a lot of lock contention. I saw a lot of lock contention in the calls to SSL_get_error() ssl3_accept(). I am running RHEL 6.5 and openssl-1.0.1e-30.el6_6.7.x86_64. We have used SSL_locking_callback() to setup pthread mutex locks, but haven’t setup dynlocks yet. Would dynlocks help in this scenario?<div class=""><br class=""></div><div class=""><div class=""><br class=""></div><div class="">Output from perf:</div><div class=""><div class="">Samples: 1M of event 'cycles', Event count (approx.): 602783506611</div>
<div class="">- 48.04% [ET_NET 0] [kernel.kallsyms] [k] _spin_lock ◆</div>
<div class="">- _spin_lock ▒</div>
<ul class=""><li class="">+ 48.05% futex_wait_setup ▒</li><li class="">+ 47.79% futex_wake ▒</li><li class="">+ 1.20% tcp_v4_rcv ▒</li><li class="">+ 0.98% task_rq_lock ▒</li><li class="">+ 0.64% double_lock_balance ▒</li></ul>
<div class="">+ 3.94% [ET_NET 0] libcrypto.so.1.0.1e [.] bn_mul_mont ▒</div>
<div class="">+ 1.93% [ET_NET 0] libcrypto.so.1.0.1e [.] BN_usub ▒</div>
<div class="">+ 1.80% [ET_NET 0] libcrypto.so.1.0.1e [.] 0x00000000000e5200 ▒</div>
<div class="">+ 1.49% [ET_NET 0] libcrypto.so.1.0.1e [.] sha256_block_data_order ▒</div>
<div class="">+ 0.97% [ET_NET 0] libcrypto.so.1.0.1e [.] gcm_ghash_clmul ▒</div>
<div class="">+ 0.85% [ET_NET 0] libpthread-2.12.so [.] pthread_mutex_lock ▒</div>
<div class="">+ 0.84% [ET_NET 0] libcrypto.so.1.0.1e [.] aesni_ctr32_encrypt_blocks ▒</div>
<div class="">+ 0.74% [ET_NET 0] libc-2.12.so [.] malloc ▒</div>
<div class="">+ 0.72% [ET_NET 0] libc-2.12.so [.] memcpy ▒</div>
<div class="">+ 0.57% [ET_NET 0] libcrypto.so.1.0.1e [.] BN_mod_mul_montgomery ▒</div>
<div class="">+ 0.51% [ET_NET 0] libpthread-2.12.so [.] pthread_mutex_unlock</div><div class=""><br class="webkit-block-placeholder"></div><div class="">pstack of the process (only a couple of the interesting threads):</div><div class=""><div class=""><span style="font-family: 'Andale Mono';" class="">Thread 1 (Thread 0x7f1fba151800 (LWP 1106)):</span></div>
<div class=""><span style="font-family: 'Andale Mono';" class="">#0 0x00007f1fbaaeb264 in __lll_lock_wait () from /lib64/libpthread.so.0</span></div>
<div class=""><span style="font-family: 'Andale Mono';" class="">#1 0x00007f1fbaae6508 in _L_lock_854 () from /lib64/libpthread.so.0</span></div>
<div class=""><span style="font-family: 'Andale Mono';" class="">#2 0x00007f1fbaae63d7 in pthread_mutex_lock () from /lib64/libpthread.so.0</span></div>
<div class=""><span style="font-family: 'Andale Mono';" class="">#3 0x000000000072822f in SSL_locking_callback(int, int, char const*, int) ()</span></div>
<div class=""><span style="font-family: 'Andale Mono';" class="">#4 0x00000031d2ae7f03 in ?? () from /usr/lib64/libcrypto.so.10</span></div>
<div class=""><span style="font-family: 'Andale Mono';" class="">#5 0x00000031d2ae7d8c in ?? () from /usr/lib64/libcrypto.so.10</span></div>
<div class=""><span style="font-family: 'Andale Mono';" class="">#6 0x00000031d2ae80ab in ERR_get_state () from /usr/lib64/libcrypto.so.10</span></div>
<div class=""><span style="font-family: 'Andale Mono';" class="">#7 0x00000031d2ae8557 in ?? () from /usr/lib64/libcrypto.so.10</span></div>
<div class=""><span style="font-family: 'Andale Mono';" class="">#8 0x00000031d2e424a9 in SSL_get_error () from /usr/lib64/libssl.so.10</span></div>
<div class=""><span style="font-family: 'Andale Mono';" class="">#9 0x00000000007249ef in SSLNetVConnection::load_buffer_and_write(long, long&, long&, MIOBufferAccessor&, int&) ()</span></div>
<div class=""><span style="font-family: 'Andale Mono';" class="">#10 0x00000000007389df in write_to_net_io(NetHandler*, UnixNetVConnection*, EThread*) ()</span></div>
<div class=""><span style="font-family: 'Andale Mono';" class="">#11 0x0000000000738523 in write_to_net(NetHandler*, UnixNetVConnection*, EThread*) ()</span></div>
<div class=""><span style="font-family: 'Andale Mono';" class="">#12 0x0000000000731922 in NetHandler::mainNetEvent(int, Event*) ()</span></div>
<div class=""><span style="font-family: 'Andale Mono';" class="">#13 0x00000000004f6d48 in Continuation::handleEvent(int, void*) ()</span></div>
<div class=""><span style="font-family: 'Andale Mono';" class="">#14 0x000000000075877a in EThread::process_event(Event*, int) ()</span></div>
<div class=""><span style="font-family: 'Andale Mono';" class="">#15 0x0000000000758c84 in EThread::ex</span></div>
<div class=""><span style="font-family: 'Andale Mono';" class=""><br class=""></span></div>
<div class=""><span style="font-family: 'Andale Mono';" class="">Thread 56 (Thread 0x7f1fb0ecd700 (LWP 1172)):</span></div>
<div class=""><span style="font-family: 'Andale Mono';" class="">#0 0x00007f1fbaae77d9 in pthread_mutex_unlock () from /lib64/libpthread.so.0</span></div>
<div class=""><span style="font-family: 'Andale Mono';" class="">#1 0x0000000000728265 in SSL_locking_callback(int, int, char const*, int) ()</span></div>
<div class=""><span style="font-family: 'Andale Mono';" class="">#2 0x00000031d2b57402 in ?? () from /usr/lib64/libcrypto.so.10</span></div>
<div class=""><span style="font-family: 'Andale Mono';" class="">#3 0x00000031d2b574bd in FIPS_selftest_failed () from /usr/lib64/libcrypto.so.10</span></div>
<div class=""><span style="font-family: 'Andale Mono';" class="">#4 0x00000031d2ae9f1c in EVP_DigestInit_ex () from /usr/lib64/libcrypto.so.10</span></div>
<div class=""><span style="font-family: 'Andale Mono';" class="">#5 0x00000031d2a7977e in HMAC_Init_ex () from /usr/lib64/libcrypto.so.10</span></div>
<div class=""><span style="font-family: 'Andale Mono';" class="">#6 0x00000031d2a79c29 in ?? () from /usr/lib64/libcrypto.so.10</span></div>
<div class=""><span style="font-family: 'Andale Mono';" class="">#7 0x00000031d2af7a42 in EVP_PKEY_CTX_ctrl () from /usr/lib64/libcrypto.so.10</span></div>
<div class=""><span style="font-family: 'Andale Mono';" class="">#8 0x00000031d2ae9ffb in EVP_DigestInit_ex () from /usr/lib64/libcrypto.so.10</span></div>
<div class=""><span style="font-family: 'Andale Mono';" class="">#9 0x00000031d2af952a in ?? () from /usr/lib64/libcrypto.so.10</span></div>
<div class=""><span style="font-family: 'Andale Mono';" class="">#10 0x00000031d2e34cf2 in tls1_PRF () from /usr/lib64/libssl.so.10</span></div>
<div class=""><span style="font-family: 'Andale Mono';" class="">#11 0x00000031d2e35796 in tls1_final_finish_mac () from /usr/lib64/libssl.so.10</span></div>
<div class=""><span style="font-family: 'Andale Mono';" class="">#12 0x00000031d2e2db2e in ssl3_get_message () from /usr/lib64/libssl.so.10</span></div>
<div class=""><span style="font-family: 'Andale Mono';" class="">#13 0x00000031d2e1dc3c in ssl3_get_cert_verify () from /usr/lib64/libssl.so.10</span></div>
<div class=""><span style="font-family: 'Andale Mono';" class="">#14 0x00000031d2e220d6 in ssl3_accept () from /usr/lib64/libssl.so.10</span></div>
<div class=""><span style="font-family: 'Andale Mono';" class="">#15 0x0000000000725287 in SSLNetVConnection::sslServerHandShakeEvent(int&) ()</span></div>
<div class=""><span style="font-family: 'Andale Mono';" class="">#16 0x00000000007251b3 in SSLNetVConnection::sslStartHandShake(int, int&) ()</span></div>
<div class=""><span style="font-family: 'Andale Mono';" class="">#17 0x0000000000723df3 in SSLNetVConnection::net_read_io(NetHandler*, EThread*) ()</span></div>
<div class=""><span style="font-family: 'Andale Mono';" class="">#18 0x0000000000731852 in NetHandler::mainNetEvent(int, Event*) ()</span></div>
<div class=""><span style="font-family: 'Andale Mono';" class="">#19 0x00000000004f6d48 in Continuation::handleEvent(int, void*) ()</span></div>
<div class=""><span style="font-family: 'Andale Mono';" class="">#20 0x000000000075877a in EThread::process_event(Event*, int) ()</span></div>
<div class=""><span style="font-family: 'Andale Mono';" class="">#21 0x0000000000758c84 in EThread::execute() ()</span></div>
<div class=""><span style="font-family: 'Andale Mono';" class="">#22 0x0000000000757cf8 in spawn_thread_internal(void*) ()</span></div>
<div class=""><span style="font-family: 'Andale Mono';" class="">#23 0x00007f1fbaae49d1 in start_thread () from /lib64/libpthread.so.0</span></div>
<div class=""><span style="font-family: 'Andale Mono';" class="">#24 0x00000030ff0e88fd in clone () from /lib64/libc.so.6</span></div></div><div class=""><br class=""></div><div class="">
<div style="color: rgb(0, 0, 0); letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class="">-Bryan</div><div class=""><br class=""></div></div><br class="Apple-interchange-newline"><br class="Apple-interchange-newline">
</div>
<br class=""></div></div></body></html>