[openssl-dev] [openssl.org #4580] "openssl verify -CAfile cacerts.pem cert.pem" fails if cacerts.pem is ordered in certain ways
Gábor STEFANIK via RT
rt at openssl.org
Tue Jun 21 01:38:21 UTC 2016
Dear OpenSSL developers,
We recently experienced an issue with our internal Mercurial repositories where Mercurial will refuse to connect to the repository due to an SSL certificate error.
The problem appeared to show up randomly on some machines, but not others.
The repository is hosted on an SCM-Manager instance and served over https with a certificate signed by our internal root CA, which is in turn deployed to the
machines on our network using Active Directory group policies.
Recently, our IT department has pushed out a new root CA certificate through Active Directory. This CA is a renewal of the old one, with the validity time extended through 2021, and the signature algorithm changed to SHA256+RSA (was SHA1+RSA before), this was when the problem started happening. The old root CA was not revoked or uninstalled.
I tracked down the issue to a problem with OpenSSL, which Mercurial uses for SSL support.
Specifically, if OpenSSL is asked to verify an SSL certificate using a CA certificate store containing both a valid CA certificate and an expired one, both correct CA certs for the SSL one, it will _sometimes_ reject the certificate with the following error message:
secondary.pem: CN = testing
error 10 at 1 depth lookup:certificate has expired
The behavior is dependent on the physical ordering of the CA certificates in the store. Specifically, if the only certificates in the store are the valid and expired CAs, then the error occurs only if the valid one comes first. With more than two certificates in the store, the behavior appears to be completely random (but shows up consistently for the same orderings).
E.g. with two valid CAs and an expired one, all 3 signing the same SSL cert, the following orders work:
older valid CA, expired CA, newer valid CA
older valid CA, newer valid CA, expired CA
newer valid CA, expired CA, older valid CA
but these fail with "certificate has expired":
expired CA, older valid CA, newer valid CA (logical ordering, this is the order in which Windows returns certificates deployed using AD)
expired CA, newer valid CA, older valid CA
newer valid CA, older valid CA, expired CA
As a simple testcase, I created the following 3 PEM files:
goodcacerts.pem:
-----BEGIN CERTIFICATE-----
MIIEwzCCAqugAwIBAgIEAwvYVTANBgkqhkiG9w0BAQsFADASMRAwDgYDVQQDEwd0
ZXN0aW5nMB4XDTA3MDcwNzA1MDcwN1oXDTA4MDcwNjA1MDcwN1owEjEQMA4GA1UE
AxMHdGVzdGluZzCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBAIHAVog+
GiK2M/QqsnCDTH4U9C2Bp47CEGNvHBWLVJh2ikfnpydmQUgo2JPeAn7UldRZKtS0
ORQt9mSvFdL9llxfbQVn4XnJM3GzKewQjrvBvBm0c9v99k37F0F0rlDrm1wWzs8C
Jzw2xiBSeznpLWCWtOKvWy7P9kGNlN32xNNw7P9hSfMjwXeXICCWlj0nD1rngoyp
jvrrobXGpZ+5gUUBg40AspLlN0bsaMjBSBmpFIZs7g9iPmVP4ANlzkPOwjLfXlk3
1CFef8F48NrlprjAzRAvSSgpixDCeYu/yjK+1IKCo1g9bYTVv41RlCs9f2Q6JjQ8
t/CtHkFnin6wKgnEQwpkwPd8DIRpgKzcYQkFym2kPolP1BPE6vvuLEwb2k1sokjQ
U35HEL9UvLzhEPlIPqtzaOcV3eQb08GpLv8CihHHubtWXV4DIrkBHwLDwC9inM/l
NSc3JefFy8I6BWxxzmXu8Wcvn9EsheJKswnJB+/0IgBuXT5RFdpBNCFRI/mdb33N
bIpS27J7CZW3sbXT2UVMuYTU7862nymd0m/bY8PQHypqep0xCCzEd35qC4jaY+RA
DJZQGRbEXfOkRGkxxzvES8m87pThT5SIJzutuiiufmbtlqdZWv+a3v/L+rpjIBet
cRF2Sb44aTmc9RvAJ802s7XxlrYuyqy4D/MzAgMBAAGjITAfMB0GA1UdDgQWBBSY
/19/f9b2cU+d4GT8hVS7LWD+ejANBgkqhkiG9w0BAQsFAAOCAgEAV+PnySD6Nkct
SsohAvra8Xuski5ZwHfymAICRM9ZWD6s/2/7y652vegMw40wxW3ZeT7NTzoi+MiT
LgYsbdu7upRGvS6dnB7ye8udOtIPhk/QPqDgH0SkxB2imeUaprX8EJUYFQX0AUd7
MAqMcqVFrpYo7xjXCxsWpWPzT6USsG9gjcSbWoukxIGmhlNvtKycSY3+L0HFZKyr
mMHZxYsUxVyPKBwfhwQTC/RQccCNLRdW68eaJSjxrbFjkMCLtppGqKHfRr/KBMvf
7K5q52vZ2O324yVdQdQjPuLk84pyJmyZ8Ew48XK06ebHgbx+fBDsNYq27IU4GRm+
FMl/Rp0q/fxZ3CphBNBdyDRRIGzIJN4e3ocSeNDcQt6uD7h88D0FrCXrlkxYGW51
qvaxFmUXlAjuv0K2iRDzvVbP2A6JhB2PBUw/lkEDcy2WP0WctLdL/GVeatRTUlzw
jxQRSi21Azwq9I4fqG/cwDDiWPGmyP2/xJoZROhky0zm/0A9mWEU9R7tB+FppZZN
3Un9HtSIRt30mqC1AgcIdiFlXpS5RiJElgIEuJVma3P2BxFK2glycy12UYy9gcOw
VV9D2OrLNxKp9icmy/D2b9Z2udLji9jI0MIhl/xDlshsTtkDmOshgqnKIyNzF/gF
8fgqQZm0AezWcLtrnbFTSG2f6Rpdjec=
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIExTCCAq2gAwIBAgIEKKekeTANBgkqhkiG9w0BAQsFADASMRAwDgYDVQQDEwd0
ZXN0aW5nMCAXDTA3MTIzMTIzMDAwMFoYDzIwOTcwOTE2MjMwMDAwWjASMRAwDgYD
VQQDEwd0ZXN0aW5nMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAgcBW
iD4aIrYz9CqycINMfhT0LYGnjsIQY28cFYtUmHaKR+enJ2ZBSCjYk94CftSV1Fkq
1LQ5FC32ZK8V0v2WXF9tBWfheckzcbMp7BCOu8G8GbRz2/32TfsXQXSuUOubXBbO
zwInPDbGIFJ7OektYJa04q9bLs/2QY2U3fbE03Ds/2FJ8yPBd5cgIJaWPScPWueC
jKmO+uuhtcaln7mBRQGDjQCykuU3RuxoyMFIGakUhmzuD2I+ZU/gA2XOQ87CMt9e
WTfUIV5/wXjw2uWmuMDNEC9JKCmLEMJ5i7/KMr7UgoKjWD1thNW/jVGUKz1/ZDom
NDy38K0eQWeKfrAqCcRDCmTA93wMhGmArNxhCQXKbaQ+iU/UE8Tq++4sTBvaTWyi
SNBTfkcQv1S8vOEQ+Ug+q3No5xXd5BvTwaku/wKKEce5u1ZdXgMiuQEfAsPAL2Kc
z+U1Jzcl58XLwjoFbHHOZe7xZy+f0SyF4kqzCckH7/QiAG5dPlEV2kE0IVEj+Z1v
fc1silLbsnsJlbextdPZRUy5hNTvzrafKZ3Sb9tjw9AfKmp6nTEILMR3fmoLiNpj
5EAMllAZFsRd86REaTHHO8RLybzulOFPlIgnO626KK5+Zu2Wp1la/5re/8v6umMg
F61xEXZJvjhpOZz1G8AnzTaztfGWti7KrLgP8zMCAwEAAaMhMB8wHQYDVR0OBBYE
FJj/X39/1vZxT53gZPyFVLstYP56MA0GCSqGSIb3DQEBCwUAA4ICAQBP1bWTxa0L
UkBHMMBvnRkof+qE7hms/mJslKFVoYxD+fRoBKU2vPPCqYqbENwMGZtHxBQ2CZFU
y4gUJQClrzqT1Pk5IcSVkAwe4RFQ7+IalITvyF0QRQZ1qaLvo893lBdQH2RfLyZG
hFl6z+dPjWBvh8w9Oo/LpzyV1LEYd25LtPL3ZJsH5Xbh1RP2oESVkd/qacxat3kn
ljmpxsuj4xJz1/VOin8xRJN97bz9kUO76Fs9ICCHSPunaVoYucCzuOG8JYPaIuUG
9I8ymNPDjcI48Gf+mxS4cs89NuSYGJ2CLB2/Knx2ViebcKnx79x62QmYco8vF5iK
G08yGJ4E4J+ERj5j7EfnUfKFDUSowzt97l7cJ1hDOx8qoeWSbaQdC2PlF5NnBZLW
mkCmMttlCs75V15Bw662AB1vGzEzKn79dTv2XsLOf7105Oq+0wgPH+iKXRYLYnDR
s+uGTY5pCU5K37Uv4T7m761JDqkipMobhWSDjzaTI4LE8qmJGWb3jwmW4nZBZmrT
FddmZyOMIQUpkLcNHea2ESdbFdo2JsUnfMGtY3XxAoeIW6+JYSBwlfjkpECM9gPH
4L27x01NmQL5yVxNHjN0EjittE/HHMXgntIRTYktjN5V8eZla85vwR72PSafUtpr
x51x19J+pGfSpXRPO48tGi8+YUckS7qN4w==
-----END CERTIFICATE-----
badcacerts.pem:
-----BEGIN CERTIFICATE-----
MIIExTCCAq2gAwIBAgIEKKekeTANBgkqhkiG9w0BAQsFADASMRAwDgYDVQQDEwd0
ZXN0aW5nMCAXDTA3MTIzMTIzMDAwMFoYDzIwOTcwOTE2MjMwMDAwWjASMRAwDgYD
VQQDEwd0ZXN0aW5nMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAgcBW
iD4aIrYz9CqycINMfhT0LYGnjsIQY28cFYtUmHaKR+enJ2ZBSCjYk94CftSV1Fkq
1LQ5FC32ZK8V0v2WXF9tBWfheckzcbMp7BCOu8G8GbRz2/32TfsXQXSuUOubXBbO
zwInPDbGIFJ7OektYJa04q9bLs/2QY2U3fbE03Ds/2FJ8yPBd5cgIJaWPScPWueC
jKmO+uuhtcaln7mBRQGDjQCykuU3RuxoyMFIGakUhmzuD2I+ZU/gA2XOQ87CMt9e
WTfUIV5/wXjw2uWmuMDNEC9JKCmLEMJ5i7/KMr7UgoKjWD1thNW/jVGUKz1/ZDom
NDy38K0eQWeKfrAqCcRDCmTA93wMhGmArNxhCQXKbaQ+iU/UE8Tq++4sTBvaTWyi
SNBTfkcQv1S8vOEQ+Ug+q3No5xXd5BvTwaku/wKKEce5u1ZdXgMiuQEfAsPAL2Kc
z+U1Jzcl58XLwjoFbHHOZe7xZy+f0SyF4kqzCckH7/QiAG5dPlEV2kE0IVEj+Z1v
fc1silLbsnsJlbextdPZRUy5hNTvzrafKZ3Sb9tjw9AfKmp6nTEILMR3fmoLiNpj
5EAMllAZFsRd86REaTHHO8RLybzulOFPlIgnO626KK5+Zu2Wp1la/5re/8v6umMg
F61xEXZJvjhpOZz1G8AnzTaztfGWti7KrLgP8zMCAwEAAaMhMB8wHQYDVR0OBBYE
FJj/X39/1vZxT53gZPyFVLstYP56MA0GCSqGSIb3DQEBCwUAA4ICAQBP1bWTxa0L
UkBHMMBvnRkof+qE7hms/mJslKFVoYxD+fRoBKU2vPPCqYqbENwMGZtHxBQ2CZFU
y4gUJQClrzqT1Pk5IcSVkAwe4RFQ7+IalITvyF0QRQZ1qaLvo893lBdQH2RfLyZG
hFl6z+dPjWBvh8w9Oo/LpzyV1LEYd25LtPL3ZJsH5Xbh1RP2oESVkd/qacxat3kn
ljmpxsuj4xJz1/VOin8xRJN97bz9kUO76Fs9ICCHSPunaVoYucCzuOG8JYPaIuUG
9I8ymNPDjcI48Gf+mxS4cs89NuSYGJ2CLB2/Knx2ViebcKnx79x62QmYco8vF5iK
G08yGJ4E4J+ERj5j7EfnUfKFDUSowzt97l7cJ1hDOx8qoeWSbaQdC2PlF5NnBZLW
mkCmMttlCs75V15Bw662AB1vGzEzKn79dTv2XsLOf7105Oq+0wgPH+iKXRYLYnDR
s+uGTY5pCU5K37Uv4T7m761JDqkipMobhWSDjzaTI4LE8qmJGWb3jwmW4nZBZmrT
FddmZyOMIQUpkLcNHea2ESdbFdo2JsUnfMGtY3XxAoeIW6+JYSBwlfjkpECM9gPH
4L27x01NmQL5yVxNHjN0EjittE/HHMXgntIRTYktjN5V8eZla85vwR72PSafUtpr
x51x19J+pGfSpXRPO48tGi8+YUckS7qN4w==
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIEwzCCAqugAwIBAgIEAwvYVTANBgkqhkiG9w0BAQsFADASMRAwDgYDVQQDEwd0
ZXN0aW5nMB4XDTA3MDcwNzA1MDcwN1oXDTA4MDcwNjA1MDcwN1owEjEQMA4GA1UE
AxMHdGVzdGluZzCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBAIHAVog+
GiK2M/QqsnCDTH4U9C2Bp47CEGNvHBWLVJh2ikfnpydmQUgo2JPeAn7UldRZKtS0
ORQt9mSvFdL9llxfbQVn4XnJM3GzKewQjrvBvBm0c9v99k37F0F0rlDrm1wWzs8C
Jzw2xiBSeznpLWCWtOKvWy7P9kGNlN32xNNw7P9hSfMjwXeXICCWlj0nD1rngoyp
jvrrobXGpZ+5gUUBg40AspLlN0bsaMjBSBmpFIZs7g9iPmVP4ANlzkPOwjLfXlk3
1CFef8F48NrlprjAzRAvSSgpixDCeYu/yjK+1IKCo1g9bYTVv41RlCs9f2Q6JjQ8
t/CtHkFnin6wKgnEQwpkwPd8DIRpgKzcYQkFym2kPolP1BPE6vvuLEwb2k1sokjQ
U35HEL9UvLzhEPlIPqtzaOcV3eQb08GpLv8CihHHubtWXV4DIrkBHwLDwC9inM/l
NSc3JefFy8I6BWxxzmXu8Wcvn9EsheJKswnJB+/0IgBuXT5RFdpBNCFRI/mdb33N
bIpS27J7CZW3sbXT2UVMuYTU7862nymd0m/bY8PQHypqep0xCCzEd35qC4jaY+RA
DJZQGRbEXfOkRGkxxzvES8m87pThT5SIJzutuiiufmbtlqdZWv+a3v/L+rpjIBet
cRF2Sb44aTmc9RvAJ802s7XxlrYuyqy4D/MzAgMBAAGjITAfMB0GA1UdDgQWBBSY
/19/f9b2cU+d4GT8hVS7LWD+ejANBgkqhkiG9w0BAQsFAAOCAgEAV+PnySD6Nkct
SsohAvra8Xuski5ZwHfymAICRM9ZWD6s/2/7y652vegMw40wxW3ZeT7NTzoi+MiT
LgYsbdu7upRGvS6dnB7ye8udOtIPhk/QPqDgH0SkxB2imeUaprX8EJUYFQX0AUd7
MAqMcqVFrpYo7xjXCxsWpWPzT6USsG9gjcSbWoukxIGmhlNvtKycSY3+L0HFZKyr
mMHZxYsUxVyPKBwfhwQTC/RQccCNLRdW68eaJSjxrbFjkMCLtppGqKHfRr/KBMvf
7K5q52vZ2O324yVdQdQjPuLk84pyJmyZ8Ew48XK06ebHgbx+fBDsNYq27IU4GRm+
FMl/Rp0q/fxZ3CphBNBdyDRRIGzIJN4e3ocSeNDcQt6uD7h88D0FrCXrlkxYGW51
qvaxFmUXlAjuv0K2iRDzvVbP2A6JhB2PBUw/lkEDcy2WP0WctLdL/GVeatRTUlzw
jxQRSi21Azwq9I4fqG/cwDDiWPGmyP2/xJoZROhky0zm/0A9mWEU9R7tB+FppZZN
3Un9HtSIRt30mqC1AgcIdiFlXpS5RiJElgIEuJVma3P2BxFK2glycy12UYy9gcOw
VV9D2OrLNxKp9icmy/D2b9Z2udLji9jI0MIhl/xDlshsTtkDmOshgqnKIyNzF/gF
8fgqQZm0AezWcLtrnbFTSG2f6Rpdjec=
-----END CERTIFICATE-----
site.pem:
-----BEGIN CERTIFICATE-----
MIIE5TCCAs2gAwIBAgIEBxm5gTANBgkqhkiG9w0BAQsFADASMRAwDgYDVQQDEwd0
ZXN0aW5nMB4XDTE2MDYyMDIwMDU1NVoXDTM4MTEyNDIwMDU1NVowEzERMA8GA1UE
AxMIdGVzdGluZzIwggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQCfeSXR
qF0VCSZ3cT6bnEP4BkVqnM09RI+Ycn/Oh5A/FGDqHc2T2OcdvkTU4KnuwPqShW5x
gvNGSeHDBHyYcHzrcu/unyVgt8u/247g4egq/Ep8edNkSkdws5N9JTxwm74H31af
ndSRYeyC2m2pwZx1sVd0gOK1NodlpP/hcm7NPrcAE6HFzkemzJrsCgxQFhWsRvba
GlEA40ZhOjGvSKfAb2oCfpW5pFYx7kSI1ii/VfX8H7CU7ZJUmj7stuc0pXZvu11v
FsssenVX7XBxBwrDDoJdjCeEz1zfl68P7QhuCLS541SgHg/ZQPSTyMsQlzhR05Z1
qaiP6oUFQSFvO7j3e6UbIgSxyXK1tRevbhwGLPBzfUgUzeKiRrMAkIDj1dC9NjK2
DAKO5VuvvQKhCLD21a5XhFMDjV1Ig2TUMMT1n55f4I6EXadcBXcBRPszZNzEgVRd
lsNGmsESCQ60bdRAnzxbbuP3A7Ufi0ASljzu3wwNGlOoEGDR3aOl341GO5x7z/gB
sMAaN0LqZnb3ACjKW6NE1jsYPWtW37y6xjnTliWA7wmcUoOZwYgwTOGuF+3lmyto
vJT2XVnli3wWUT2TPh9t4MgGzYePYiinpem3S9Ylsw7JNR0r4EDqyLF+8r4+Wg2V
AhpOKkzcOfRaxXjxWNiJ8CqLpb6E8P7p2cb1QwIDAQABo0IwQDAfBgNVHSMEGDAW
gBSY/19/f9b2cU+d4GT8hVS7LWD+ejAdBgNVHQ4EFgQU/+XByYexoutQKmN8aKwm
9ihiokYwDQYJKoZIhvcNAQELBQADggIBADX9+ihl9fGZelPZ426NYBWHvhr2LjB2
gVVPQpGG53AD5HOSRD1Lf0atfOXWnM0DXqN15cEp4jhB5ui16Vp8dYbU+7TSMpjX
W3emVBk09oHgYA23W8OSdi93rx5JARGKY36aY1Si79MnJ40mR3LlZqnx54IOujX0
k8kLEkZ+MzbZnxxYfupr8wqihAgpNFJCTuW7QT9tlHATdlUN1zsdh9RtPal4Wngi
dKk/L02VvvCLLfs37HCf8X/D89YDCH6XvXYok2TdhOe0VIiaa1JB0e0G/4rraPqw
e80FRr+5tEJFRMgG0QYgZo61l73UMNNB0MQ6xKN0UR6K9W05B7ILwAMiRyZFCJ9f
E+voJix+IWNtb495XfQGKpc/ALK+ifCdo1s6gIdE3RtEMZAwkdDjut/wtjaDfH78
8HUYJwAyrfQERrX5InF6pC9i5GasUcFnrq3zgq2jjddNMCQVbPZYxiabXq0qwDho
Mevk5tkPunZEfE3Npbb85vxPnn7z1uE3C6CDr0ZIRC2frlD6ShFqw6FhFQYnNvXd
Phs8fKQEwPN/PSb46rY/iLbASMNvh14Ut971Ta4ZKW/JHemfeRPUzqCECbE0jiKq
k5KEX5Ge7lLQ6i1q6jEPizR9z/HPbkkSNEIJN+UaTXW78i+0WGP8cLQIzGZFRNZB
uxRMPjdZ/XpR
-----END CERTIFICATE-----
goodcacerts.pem and badcacerts.pem contain exactly the same certificates (only ordered differently), yet:
C:\Users\gstefanik\Documents\buggycerts>openssl verify -CAfile goodcacerts.pem site.pem
site.pem: OK
C:\Users\gstefanik\Documents\buggycerts>openssl verify -CAfile badcacerts.pem site.pem
site.pem: CN = testing
error 10 at 1 depth lookup:certificate has expired
OK
Even worse, some 32-bit Windows builds of openssl actually crash when verifying using badcacerts.pem (e.g. https://indy.fulgan.com/SSL/openssl-1.0.2h-i386-win32.zip)
Unfortunately I was not able to compile a 32-bit Windows build myself.
Sincerely,
Gábor Stefanik
--------------------------------------------------------------------------
This message, including its attachments, is confidential. For more information please read NNG's email policy here:
http://www.nng.com/emailpolicy/
By responding to this email you accept the email policy.
--
Ticket here: http://rt.openssl.org/Ticket/Display.html?id=4580
Please log in as guest with password guest if prompted
More information about the openssl-dev
mailing list