<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 12 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:SimSun;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:SimSun;
panose-1:2 1 6 0 3 1 1 1 1 1;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
text-align:justify;
text-justify:inter-ideograph;
font-size:10.5pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;}
/* Page Definitions */
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 90.0pt 72.0pt 90.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="ZH-CN" link="blue" vlink="purple" style="text-justify-trim:punctuation">
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-US">Hi, all<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">There is a probability error when I update the ovsdb-server ca_cert.pem file, the ovsdb-client was unable to connect to the ovsdb-server when it hanppened, the OVS version is 2.0.2.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">the update action steps on server:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">step1: rm ca_cert.pem<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">step2: openssl x509 -inform PEM -in ca_cert.pem > /home/ca_cert.pem<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">the server update script :<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">#!/bin/bash<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">SRC_CA_CRT_FILE=/home/pem/oam-network-agent_ca_crt.pem<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">DST_CA_CRT_FILE=/home/oam-network-agent_ca_crt.pem<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">for((i=0; i<50000; i++));do<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">rm -f $DST_CA_CRT_FILE<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">sleep 0.5<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">openssl x509 -inform PEM -in $SRC_CA_CRT_FILE > /home/ca_crt.pem.tmp<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">mv /home/ca_crt.pem.tmp $DST_CA_CRT_FILE<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">echo "update-----result $?-------------------------------------$i"<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">done<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">the client connect script:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">for((i=0; i<50000; i++));do<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> ovsdb-client -v -p /home/oam-network-agent_private_key.pem -c oam-network-agent_crt.pem -C /home/oam-network-agent_ca_crt.pem get-schema ssl:9.42.3.9:6632 Open_vSwitch<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> sleep 0.5<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> echo $i<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">done<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">running server update script and client connect script on the sametime, after a period of time, the ovsdb-client can not connect the server, the error like ERROR1 and ERROR2.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">I have found the direct cause is concurrency write-read file issues, the ovsdb-server probably read the wrong certificate from ca_cert.pem file, but this error is unrecoverable, it need to restart OVS to fix,
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">did someone know about this problem?<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">The ovsdb-client connected error like this:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">ERROR1:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"># ovsdb-client -v -p /home/oam-network-agent_private_key.pem -c oam-network-agent_crt.pem -C /home/oam-network-agent_ca_crt.pem get-schema ssl:9.42.3.9:6632 Open_vSwitch<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">2015-09-25T10:54:36Z|00001|stream_ssl|INFO|Trusting CA cert from /home/oam-network-agent_ca_crt.pem (/C=CN/ST=ZheJiang/O=Huawei/OU=Huawei/CN=*.*.*.domainname.com) (fingerprint 22:a3:49:97:e1:44:ab:fb:96:29:60:ab:b8:fc:69:8b:7d:af:6c:6e)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">2015-09-25T10:54:36Z|00002|poll_loop|DBG|wakeup due to 0-ms timeout<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">2015-09-25T10:54:36Z|00003|poll_loop|DBG|wakeup due to [POLLOUT] on fd 4 (9.62.243.149:54185<->9.42.3.9:6632) at lib/stream-ssl.c:716<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">2015-09-25T10:54:36Z|00004|stream_ssl|DBG|client0-->ssl:9.42.3.9:6632 handshake: client_hello (85 bytes)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">2015-09-25T10:54:36Z|00005|poll_loop|DBG|wakeup due to [POLLIN] on fd 4 (9.62.243.149:54185<->9.42.3.9:6632) at lib/stream-ssl.c:723<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">2015-09-25T10:54:36Z|00006|stream_ssl|DBG|client0<--ssl:9.42.3.9:6632 handshake: server_hello (53 bytes)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">2015-09-25T10:54:36Z|00007|stream_ssl|DBG|client0<--ssl:9.42.3.9:6632 handshake: certificate (1944 bytes)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">2015-09-25T10:54:36Z|00008|poll_loop|DBG|wakeup due to [POLLIN] on fd 4 (9.62.243.149:54185<->9.42.3.9:6632) at lib/stream-ssl.c:723<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">2015-09-25T10:54:36Z|00009|poll_loop|DBG|wakeup due to [POLLIN] on fd 4 (9.62.243.149:54185<->9.42.3.9:6632) at lib/stream-ssl.c:723<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">2015-09-25T10:54:36Z|00010|poll_loop|DBG|wakeup due to [POLLIN] on fd 4 (9.62.243.149:54185<->9.42.3.9:6632) at lib/stream-ssl.c:723<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">2015-09-25T10:54:36Z|00011|poll_loop|DBG|wakeup due to [POLLIN] on fd 4 (9.62.243.149:54185<->9.42.3.9:6632) at lib/stream-ssl.c:723<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">2015-09-25T10:54:36Z|00012|poll_loop|DBG|wakeup due to [POLLIN] on fd 4 (9.62.243.149:54185<->9.42.3.9:6632) at lib/stream-ssl.c:723<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">2015-09-25T10:54:36Z|00013|poll_loop|DBG|wakeup due to [POLLIN] on fd 4 (9.62.243.149:54185<->9.42.3.9:6632) at lib/stream-ssl.c:723<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">2015-09-25T10:54:36Z|00014|poll_loop|DBG|wakeup due to [POLLIN] on fd 4 (9.62.243.149:54185<->9.42.3.9:6632) at lib/stream-ssl.c:723<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">2015-09-25T10:54:36Z|00015|poll_loop|DBG|wakeup due to [POLLIN] on fd 4 (9.62.243.149:54185<->9.42.3.9:6632) at lib/stream-ssl.c:723<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">2015-09-25T10:54:36Z|00016|poll_loop|DBG|wakeup due to [POLLIN] on fd 4 (9.62.243.149:54185<->9.42.3.9:6632) at lib/stream-ssl.c:723<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">2015-09-25T10:54:36Z|00017|poll_loop|DBG|wakeup due to [POLLIN] on fd 4 (9.62.243.149:54185<->9.42.3.9:6632) at lib/stream-ssl.c:723<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">2015-09-25T10:54:36Z|00048|stream_ssl|DBG|client0<--ssl:9.42.3.9:6632 handshake: certificate_request (65559 bytes)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">2015-09-25T10:54:36Z|00049|stream_ssl|DBG|client0-->ssl:9.42.3.9:6632 alert: fatal, decode_error (2 bytes)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">2015-09-25T10:54:36Z|00050|stream_ssl|WARN|SSL_connect: error:1408709F:SSL routines:SSL3_GET_CERTIFICATE_REQUEST:length mismatch<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">ovsdb-client: failed to connect to "ssl:9.42.3.9:6632" (Protocol error)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">ERROR2:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"># ovsdb-client -v -p /home/oam-network-agent_private_key.pem -c oam-network-agent_crt.pem -C /home/oam-network-agent_ca_crt.pem get-schema ssl:9.42.3.9:6632 Open_vSwitch<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">2015-09-25T11:01:06Z|00001|stream_ssl|INFO|Trusting CA cert from /home/oam-network-agent_ca_crt.pem (/C=CN/ST=ZheJiang/O=Huawei/OU=Huawei/CN=*.*.*.domainname.com) (fingerprint 22:a3:49:97:e1:44:ab:fb:96:29:60:ab:b8:fc:69:8b:7d:af:6c:6e)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">2015-09-25T11:01:06Z|00002|poll_loop|DBG|wakeup due to 0-ms timeout<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">2015-09-25T11:01:06Z|00003|poll_loop|DBG|wakeup due to [POLLOUT] on fd 4 (9.62.243.149:54288<->9.42.3.9:6632) at lib/stream-ssl.c:716<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">2015-09-25T11:01:06Z|00004|stream_ssl|DBG|client0-->ssl:9.42.3.9:6632 handshake: client_hello (85 bytes)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">2015-09-25T11:01:06Z|00005|poll_loop|DBG|wakeup due to [POLLIN] on fd 4 (9.62.243.149:54288<->9.42.3.9:6632) at lib/stream-ssl.c:723<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">2015-09-25T11:01:06Z|00006|stream_ssl|DBG|client0<--ssl:9.42.3.9:6632 handshake: server_hello (53 bytes)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">2015-09-25T11:01:06Z|00007|stream_ssl|DBG|client0<--ssl:9.42.3.9:6632 handshake: certificate (985 bytes)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">2015-09-25T11:01:06Z|00008|poll_loop|DBG|wakeup due to [POLLIN] on fd 4 (9.62.243.149:54288<->9.42.3.9:6632) at lib/stream-ssl.c:723<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">2015-09-25T11:01:06Z|00009|poll_loop|DBG|wakeup due to [POLLIN] on fd 4 (9.62.243.149:54288<->9.42.3.9:6632) at lib/stream-ssl.c:723<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">2015-09-25T11:01:06Z|00010|poll_loop|DBG|wakeup due to [POLLIN] on fd 4 (9.62.243.149:54288<->9.42.3.9:6632) at lib/stream-ssl.c:723<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">2015-09-25T11:01:06Z|00011|stream_ssl|DBG|client0<--ssl:9.42.3.9:6632 handshake: certificate_request (11019 bytes)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">2015-09-25T11:01:06Z|00012|stream_ssl|DBG|client0<--ssl:9.42.3.9:6632 handshake: server_hello_done (4 bytes)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">2015-09-25T11:01:06Z|00013|stream_ssl|DBG|client0-->ssl:9.42.3.9:6632 handshake: certificate (1944 bytes)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">2015-09-25T11:01:06Z|00014|stream_ssl|DBG|client0-->ssl:9.42.3.9:6632 handshake: client_key_exchange (262 bytes)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">2015-09-25T11:01:06Z|00015|stream_ssl|DBG|client0-->ssl:9.42.3.9:6632 handshake: certificate_verify (262 bytes)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">2015-09-25T11:01:06Z|00016|stream_ssl|DBG|client0-->ssl:9.42.3.9:6632 change_cipher_spec (1 bytes)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">2015-09-25T11:01:06Z|00017|stream_ssl|DBG|client0-->ssl:9.42.3.9:6632 handshake: finished (16 bytes)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">2015-09-25T11:01:06Z|00018|poll_loop|DBG|wakeup due to [POLLIN] on fd 4 (9.62.243.149:54288<->9.42.3.9:6632) at lib/stream-ssl.c:723<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">2015-09-25T11:01:06Z|00019|stream_ssl|DBG|client0<--ssl:9.42.3.9:6632 alert: fatal, unknown_ca (2 bytes)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">2015-09-25T11:01:06Z|00020|stream_ssl|WARN|SSL_connect: error:14094418:SSL routines:SSL3_READ_BYTES:tlsv1 alert unknown ca<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">ovsdb-client: failed to connect to "ssl:9.42.3.9:6632" (Protocol error)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">the ovsdb-server log will print warning like this:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">ERROR1:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">2015-09-25T11:05:15.633Z|02941|stream_ssl|WARN|SSL_accept: error:1409441A:SSL routines:SSL3_READ_BYTES:tlsv1 alert decode error<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">2015-09-25T11:05:15.633Z|02942|jsonrpc|WARN|ssl:9.62.243.149:54187: receive error: Protocol error<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">2015-09-25T11:05:15.634Z|02943|reconnect|WARN|ssl:9.62.243.149:54187: connection dropped (Protocol error)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">ERROR2:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">2015-09-25T11:11:37.494Z|00449|stream_ssl|WARN|SSL_accept: error:140890B2:SSL routines:SSL3_GET_CLIENT_CERTIFICATE:no certificate returned<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">2015-09-25T11:11:37.494Z|00450|jsonrpc|WARN|ssl:9.62.243.149:54289: receive error: Protocol error<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">2015-09-25T11:11:37.494Z|00451|reconnect|WARN|ssl:9.62.243.149:54289: connection dropped (Protocol error)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
</body>
</html>