/etc/ssl/certs/ca-certificates.crt interface questions

[Roy Wellington]

Hi,

I am working with a software vendor, and we're investigating what I 
currently believe is a bug in their software.

Their software is an HTTPS server. We provide the cert chain that it 
uses to serve. Internally, it makes HTTPS requests to itself, so in that 
regard, it also acts as a client, and will validate its own certificate 
at that point. During that internal request, the client currently fails 
to validate the server's certificate, and that is the context of the 
problem we're investigating.

Currently, we supply a Let's Encrypt leaf cert & chain. Currently, we 
provide Let's Encrypt's "default chain"; that is, we take what LE gives 
us, and provide to this piece of software, so that it can serve under 
the name on the cert. We are aware of the alternate chain, but for 
simplicity's sake, I'd like to avoid it, if possible. The default chain 
LE issues is leaf -> R3 intermediate -> ISRG root (cross signed by DST). 
The cert that signs that last cert, the DST root, is now expired. 
However, the IRSG root itself should be in any trust store.

As mentioned, when openssl s_client in this vendor's product attempts to 
verify the leaf (which is issues by LE), it fails:

     Verify return code: 10 (certificate has expired)

While I know old versions of OpenSSL were affected by bugs that resulted 
in that error, the vendor here is not using a particularly old version:

# openssl version
OpenSSL 1.1.1l  24 Aug 2021

In fact, normally, this version of OpenSSL appears to correctly validate 
our leaf certificate. However, this vendor also does the following:

  * they append our entire cert chain, leaf, intermediate and all, to
    /etc/ssl/certs/ca-certificates.crt
  * they append an internal, self-signed CA to
    /etc/ssl/certs/ca-certificates.crt

If I remove either just that self-signed CA, or all 4 certs (my chain + 
the self signed) from ca-certificates.crt, then s_client shows that 
validation passes:

     Verify return code: 0 (ok)

This is bizarre, to me, but in particular, that just removing an 
unrelated self-signed CA causes validation of a leaf not signed by that 
CA to succeed is /really/ weird.

My point of view to the vendor has been that appending to 
ca-certificates.crt is not valid; that instead, they should emit//root 
certs to something like /usr/local/share/ca-certificates & run 
update-ca-certificates, and that the "interface" of this directory 
includes also setting up the symlinks & hashed symlinks appropriately 
for additional CAs. (And that it is my understanding that 
update-ca-certificates handles this process for oneself automatically.) 
That is, by merely appending to ca-certificates.crt we're invoking some 
sort of undefined behavior from the library. If this is documented 
somewhere, that'd be helpful, I think, in convincing the vendor.

That said, the observed behavior is just weird, and I'd love a deeper 
explanation of why removing a self-signed cert that seems 
inconsequential from ca-certificates.crt appears to cause path building 
to reach a different conclusion. Naïvely, removing a trust anchor 
shouldn't cause /more/ leaf certs to validate, and that smells 
potentially like a bug in OpenSSL, excepting potentially UB from the 
above (ab)use of ca-certificates.crt.

Thanks for any help,
—Roy
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mta.openssl.org/pipermail/openssl-users/attachments/20211116/a993eca6/attachment.html>