/etc/ssl/certs/ca-certificates.crt interface questions
roy at commure.com
Tue Nov 16 19:21:03 UTC 2021
I am working with a software vendor, and we're investigating what I
currently believe is a bug in their software.
Their software is an HTTPS server. We provide the cert chain that it
uses to serve. Internally, it makes HTTPS requests to itself, so in that
regard, it also acts as a client, and will validate its own certificate
at that point. During that internal request, the client currently fails
to validate the server's certificate, and that is the context of the
problem we're investigating.
Currently, we supply a Let's Encrypt leaf cert & chain. Currently, we
provide Let's Encrypt's "default chain"; that is, we take what LE gives
us, and provide to this piece of software, so that it can serve under
the name on the cert. We are aware of the alternate chain, but for
simplicity's sake, I'd like to avoid it, if possible. The default chain
LE issues is leaf -> R3 intermediate -> ISRG root (cross signed by DST).
The cert that signs that last cert, the DST root, is now expired.
However, the IRSG root itself should be in any trust store.
As mentioned, when openssl s_client in this vendor's product attempts to
verify the leaf (which is issues by LE), it fails:
Verify return code: 10 (certificate has expired)
While I know old versions of OpenSSL were affected by bugs that resulted
in that error, the vendor here is not using a particularly old version:
# openssl version
OpenSSL 1.1.1l 24 Aug 2021
In fact, normally, this version of OpenSSL appears to correctly validate
our leaf certificate. However, this vendor also does the following:
* they append our entire cert chain, leaf, intermediate and all, to
* they append an internal, self-signed CA to
If I remove either just that self-signed CA, or all 4 certs (my chain +
the self signed) from ca-certificates.crt, then s_client shows that
Verify return code: 0 (ok)
This is bizarre, to me, but in particular, that just removing an
unrelated self-signed CA causes validation of a leaf not signed by that
CA to succeed is /really/ weird.
My point of view to the vendor has been that appending to
ca-certificates.crt is not valid; that instead, they should emit//root
certs to something like /usr/local/share/ca-certificates & run
update-ca-certificates, and that the "interface" of this directory
includes also setting up the symlinks & hashed symlinks appropriately
for additional CAs. (And that it is my understanding that
update-ca-certificates handles this process for oneself automatically.)
That is, by merely appending to ca-certificates.crt we're invoking some
sort of undefined behavior from the library. If this is documented
somewhere, that'd be helpful, I think, in convincing the vendor.
That said, the observed behavior is just weird, and I'd love a deeper
explanation of why removing a self-signed cert that seems
inconsequential from ca-certificates.crt appears to cause path building
to reach a different conclusion. Naïvely, removing a trust anchor
shouldn't cause /more/ leaf certs to validate, and that smells
potentially like a bug in OpenSSL, excepting potentially UB from the
above (ab)use of ca-certificates.crt.
Thanks for any help,
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the openssl-users