OpenSSL not accepting a certificate, whilst curl does.

[John Robson]

tl;dr: Found an issue with update-ca-trust extract
OpenSSL doing what it should, but update-ca-trust is only pushing the cert
into some of the trust stores.

Thanks Tomas

On Tue, 29 Sep 2020 at 07:06, Tomas Mraz <tmraz at redhat.com> wrote:
>
> On Mon, 2020-09-28 at 22:35 +0100, John Robson via openssl-users wrote:
> > Hi,
> >
> > I'm really struggling to get my head around a specific scenario that
> > isn't behaving as I expect.  Hopefully someone with more
> > experience/knowledge can set me on the right path.
> >
> > Note - my attempts to reproduce this in a lab have been unsuccessful,
> > although I don't have access to the server private key, so the
> > attempts have been with a completely independent CA chain.
> >
> >
> > I have a private CA, which has signed an intermediate certificate
> > which has signed a server certificate for an internal web server
> > which is used by various automated systems (all linux based).
> >
> > The webserver (Apache) has the server cert and key, defined and in
> > use as well as the intermediate certificate defined as the chain
> > certificate - this all shows as expected.
> >
> > I have then added the root certificate to the trusted certs for an
> > automated system (populated `/etc/pki/ca-trust/source/anchors/` run
> > `update-ca-trust extract`).
> >
> > After this curl no longer complains about the certificate from the
> > web server (expected).
> > However OpenSSL still does (unexpected), and I presume that for the
> > same reason(s) urllib in Python also doesn't accept the certificate.
> > If I manually feed `openssl verify` the certificates and chain then
> > they all come back "OK".
> >
> > I've set up these systems a number of times with both self signed and
> > CA signed certs and never seen this behaviour.
>
> ...
> >
> > OpenSSL:
> > # openssl s_client -connect server.fqdn:443
> > CONNECTED(00000007)
> > depth=1 CN = CAINTER, O = org, C = XX
> > verify error:num=2:unable to get issuer certificate
> > issuer= CN = CAROOT, O = org, C = XX
> >  --8<--
> > Verify return code: 2 (unable to get issuer certificate)


> What is the curl library linked to? Is it using OpenSSL or something
> else, for example NSS, as the TLS library? What exact system are you
> testing on?

Running curl -v on a lab box it shows that curl is using NSS, which at
least gives some reason for the different behaviour.
The system is a dockerised monitoring utility (clue is in the email
address) with the containers based on CentOS.

> Are you sure you've put your CAROOT certificate to the system
> certificate trust store? And/or is the trust store properly set up to
> be loaded by OpenSSL by default?

Curl complains about the certificate until I add the cert to the ca-trust,
so I have a good degree of confidence that the cert has been installed into
the trust store - and on all the systems I've installed thus far this has
also been picked up by OpenSSL.

openssl connect does negotiate a key and cipher, and there is a connection
created, so I'm confident there isn't a key/cipher capability mismatch,
it's just the certificate verification that is failing.

Checking where OpenSSL is using for it's CA trust store:
# openssl version -d
OPENSSLDIR: "/etc/pki/tls"

In that directory cert.pem is a symlink to
/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem, which is subtly
different from the /etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt
I had been searching earlier - however I still see the ROOTCA entry in the
pem bundle (and they have identical timestamps, indicating that both were
updated by the update-ca-trust extract command).

I'm getting a closer look at those files to make sure that the cert has
been correctly pushed into them. I can't imagine why it wouldn't have been,
but it is the next logical thing to check.

OK - The issue is that update-ca-trust extract command i was using doesn't
do what I would expect from the docs (and previous experience)

Even with the Root CA in the anchors directory it doesn't add it to the TLS
ca store, it does add it elsewhere though.
If I manually add it to the ca-bundle.trust.crt then OpenSSL does verify
the cert correctly, but the cert gets pulled if I then run the update
again, so that's not a good long term solution.

Since I have the Root CA I can test adding it to a few different containers
and see whether there is something about that cert that is causing the odd
behaviour.

Thanks to Tomas for reminding me there are various different cert trust
stores, and for ensuring that I followed the correct path deep enough to
figure out what was going on.

Now to try and work out ca-certificate behaviour,

Cheers,

John
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mta.openssl.org/pipermail/openssl-users/attachments/20200929/fe4d8726/attachment.html>