[openssl-users] Failure using ECDH-RSA-AES256-SHA with ssl3 on Master Branch

[Linsell, StevenX]

I am trying to use ECDH-RSA-AES256-SHA with ssl3 with s_client and s_server on the master branch. (cloned at commit f7683aaf36341dc65672ac2ccdbfd4a232e3626d) and then retested  with a more recent clone: (commit da27006df06853a33b132133699a7aa9d4277920).
We are running a test suite that tests all supported cipher and protocol combinations and this test is part of that suite.
Our test suite is failing with an unmodified build of OpenSSL with the following commands:-

s_server:
./openssl s_server -cert prime256v1-rsaTestServer.cert.pem -key prime256v1-rsaTestServer.key.pem -WWW -accept 4411 -cipher ECDH-RSA-AES256-SHA -nbio -ssl3 -debug -state

s_client:
echo "GET /file_1byte.html HTTP/1.0" | ./openssl s_client  -host localhost -port 4411 -cipher ECDH-RSA-AES256-SHA -ssl3 -ign_eof -debug -state

The output from s_client is:-

SSL_connect:before/connect initialization
SSL_connect:SSLv3 write client hello A
SSL3 alert read:fatal:handshake failure
SSL_connect:failed in SSLv3 read server hello A
139749978326688:error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:s3_pkt.c:1482:SSL alert number 40
139749978326688:error:1409E0E5:SSL routines:ssl3_write_bytes:ssl handshake failure:s3_pkt.c:664:
CONNECTED(00000003)
write to 0x1284120 [0x128e913] (52 bytes => 52 (0x34))
0000 - 16 03 00 00 2f 01 00 00-2b 03 00 af 73 f8 85 b4   ..../...+...s...
0010 - 01 5f d4 79 66 4e 94 fa-bf e7 5e 5b 19 75 c8 5f   ._.yfN....^[.u._
0020 - 44 73 bb bd 47 8c 23 57-01 c0 1a 00 00 04 c0 0f   Ds..G.#W........
0030 - 00 ff 01                                          ...
0034 - <SPACES/NULS>
read from 0x1284120 [0x128a3c3] (5 bytes => 5 (0x5))
0000 - 15 03 00 00 02                                    .....
read from 0x1284120 [0x128a3c8] (2 bytes => 2 (0x2))
0000 - 02 28                                             .(
---
no peer certificate available

The output from s_server is:-

Using default temp DH parameters
ACCEPT
turning on non blocking io
SSL_accept:before/accept initialization
read from 0x21b32b0 [0x21b7993] (5 bytes => 5 (0x5))
0000 - 16 03 00 00 2f                                    ..../
read from 0x21b32b0 [0x21b7998] (47 bytes => 47 (0x2F))
0000 - 01 00 00 2b 03 00 aa 75-39 f4 b5 78 46 3e 8c cb   ...+...u9..xF>..
0010 - a9 18 92 01 cd 24 cf fd-7b a7 de 29 7c b8 d9 bc   .....$..{..)|...
0020 - c4 62 1c c5 33 7f 00 00-04 c0 0f 00 ff 01         .b..3.........
002f - <SPACES/NULS>
0:[00000020:00000010:00000188:00000084]0x6055a0:ECDH-RSA-AES256-SHA
write to 0x21b32b0 [0x21c6910] (7 bytes => 7 (0x7))
0000 - 15 03 00 00 02 02 28                              ......(
SSL3 alert write:fatal:handshake failure
SSL_accept:error in SSLv3 read client hello C
139792107542176:error:1408A0C1:SSL routines:ssl3_get_client_hello:no shared cipher:s3_srvr.c:1366:
ACCEPT

I am using an ECC test certificate that uses curve prime256v1 and is signed with an rsa2k key.
The cert/key were generated using RSAcertgen.sh followed by ECC-RSAcertgen.sh modified only for the curve and RSA key size I am using.
Here is a dump of the certificate:
./openssl x509 -in prime256v1-rsaTestServer.cert.pem -text -noout
Certificate:
    Data:
        Version: 1 (0x0)
        Serial Number: 16838786626002069798 (0xe9af63387b73a926)
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=US, ST=CA, L=Mountain View, O=Sun Microsystems, Inc., OU=Sun Microsystems Laboratories, CN=Test CA (2048 bit RSA)
        Validity
            Not Before: Mar 13 11:38:21 2015 GMT
            Not After : Apr 21 11:38:21 2019 GMT
        Subject: C=US, ST=CA, L=Mountain View, O=Sun Microsystems, Inc., OU=Sun Microsystems Laboratories, CN=Test Server (prime256v1 key signed with RSA)
        Subject Public Key Info:
            Public Key Algorithm: id-ecPublicKey
                Public-Key: (256 bit)
                pub: 
                    04:0d:a6:16:d8:43:25:dc:83:6d:18:fb:f0:b7:41:
                    bc:05:88:a2:f2:56:8a:76:7a:d0:2b:7f:de:0a:44:
                    33:4b:de:5b:30:44:ff:34:0e:17:c6:38:77:d7:53:
                    b2:c2:fa:9f:7f:d5:e3:a4:b5:de:ce:29:9d:74:e6:
                    59:76:9f:e6:eb
                ASN1 OID: prime256v1
                NIST CURVE: P-256
    Signature Algorithm: sha256WithRSAEncryption
         d0:1c:97:60:b9:14:cf:5a:c8:ea:8d:65:63:75:50:f2:63:68:
         82:06:0c:47:f5:52:13:a5:61:4b:cd:99:ab:d0:56:81:a7:92:
         21:c7:07:e3:12:25:4a:a8:c7:83:7a:bd:57:11:c7:55:88:28:
         74:f1:37:bb:cd:0b:5b:7b:6f:45:e6:8d:1a:be:1a:fd:e0:d2:
         5b:e5:ee:39:2e:73:c8:d6:03:5c:f6:f9:37:4a:81:e4:41:5a:
         87:d5:0d:da:48:67:14:bb:75:3b:ae:68:b9:c4:25:2d:19:a7:
         05:90:a2:fb:b4:d3:00:4f:40:19:e9:2d:83:75:db:3c:53:fe:
         08:ae:ca:ba:3d:a5:4d:6e:f6:14:af:ee:7e:6d:dc:45:96:91:
         92:6d:37:52:b6:b7:ad:70:02:d0:11:0d:84:1b:f1:3b:82:be:
         66:af:a6:3c:17:33:d0:98:c3:cb:d3:22:39:d1:66:6e:94:ce:
         7e:70:3c:02:29:6a:b6:87:e9:c4:e9:44:b4:9b:f1:8e:47:82:
         2d:20:79:0e:f6:91:b1:e9:cf:83:66:8f:ff:e1:4f:2f:a1:ab:
         ca:2d:81:53:7d:7f:69:b5:11:59:7e:9a:47:1c:6a:c8:83:54:
         83:0a:7d:46:ec:2e:e9:82:f3:b4:d4:f6:04:57:bc:a5:b2:c5:
         0c:ed:a6:fa

Running the exact same s_server/s_client commands above with either the system openssl (1.0.0o) or the baseline we normally release against (1.0.1l) works fine.
Running on the master branch with the same certificate and commands above but with tls1, tls1_1 or tls1_2 works perfectly, only ssl3 fails.
Running with a sect163r1 curve signed with an rsa1k key also produces the same failure.
My build is as follows:
./openssl version -a
OpenSSL 1.1.0-dev xx XXX xxxx
built on: reproducible build, date unspecified
platform: linux-x86_64
options:  bn(64,64) rc4(16x,int) des(idx,cisc,16,int) idea(int) blowfish(idx) 
compiler: gcc -g -I. -I.. -I../include  -DOPENSSL_TLS_SECURITY_LEVEL=0 -DOPENSSL_THREADS -D_REENTRANT -DDSO_DLFCN -DHAVE_DLFCN_H -Wa,--noexecstack -m64 -DL_ENDIAN -O3 -Wall -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DMD5_ASM -DAES_ASM -DVPAES_ASM -DBSAES_ASM -DWHIRLPOOL_ASM -DGHASH_ASM -DECP_NISTZ256_ASM
OPENSSLDIR: "/usr/local/ssl"

As you can see the only flag I have enabled (apart from -g while debugging) is -DOPENSSL_TLS_SECURITY_LEVEL=0. We need this flag to allow some of the older cipher suites we test against but the issue is seen with or without that flag defined.

I'm not overly familiar with the master branch as we do not normally build against it so my real question is whether I am doing something wrong in terms of configuration on the master branch (is there a flag I need to enable to allow ECDH-RSA with ssl3 that I haven't spotted?) or is this a genuine bug?

Single stepping through the code I can see the failure is occurring in tls1_check_ec_key when it is called from tls1_check_cert_param.
It appears to go around a for loop (j) twice. The first time through it correctly matches the curve it is looking for. The second time round the list is empty and 0 is returned. This failure causes the Elliptical curve cert not to be declared as valid and consequently the handshake fails with the no shared cipher message.
I don't have a good understanding of how the certificate code works so I haven't managed to debug any further than that in order to determine why the second time round the loop the list is empty.

-- 
Steve Linsell                                     Intel Shannon DCG/CID Software Development Team
Stevenx.Linsell at intel.com                              


--------------------------------------------------------------
Intel Shannon Limited
Registered in Ireland
Registered Office: Collinstown Industrial Park, Leixlip, County Kildare
Registered Number: 308263
Business address: Dromore House, East Park, Shannon, Co. Clare

This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.