Strange problem: openssl verify not working on Proxmox VM, works on a bare metal system

[Shawn Heisey]

On 9/3/22 16:07, Viktor Dukhovni wrote:
> Post the output of:
>
> $ openssl crl2pkcs7 -nocrl -certfile 
> /etc/ssl/certs/local/DOMAIN.wildcards.pem |
> openssl pkcs7 -print_certs -noout |
> perl -ne 'BEGIN{$/="\n\n\n"} s/\n+/\n/g; print $_, "\n"'

subject=CN = DOMAIN
issuer=C = US, O = Let's Encrypt, CN = R3

subject=C = US, O = Let's Encrypt, CN = R3
issuer=C = US, O = Internet Security Research Group, CN = ISRG Root X1

> The verify(1) command will attempt to construct a chain to a trusted
> root using the specified or default CAfile and CApath. You should
> really be using the "-untrusted" option not the "-CAfile" option:
>
> # cert=/etc/ssl/certs/local/DOMAIN.wildcards.pem
> # openssl verify -untrusted "$cert" "$cert"
>
> This adds the untrusted intermediate certs from the cert file to
> the dataset, without shadowing the default CAfile.

One of the things my install script for quictls does is this:

         rm -rf /opt/quictls/ssl/certs
         ln -s /etc/ssl/certs /opt/quictls/ssl/certs

I do this because that certs directory installed with quictls is empty.

There is a big difference in what is in /etc/ssl/certs on Ubuntu 
compared to AlmaLinux and probably every other RPM distro.  Alma appears 
to have a ca-certificates package from Feb 2021 while Ubuntu has one 
from Nov 2021.

The combined cert file that I have built does NOT contain the root 
cert.  I only need to send the server cert and the issuing cert. The 
browser will have the root cert, so there is no need to send it.

If I use a file that DOES contain the root cert (one of the files 
downloaded by certbot) then I can get it to pass with -untrusted, but 
not -CAfile:

[root at certs ~]# openssl verify -untrusted 
/etc/letsencrypt/live/DOMAIN/chain.pem 
/etc/letsencrypt/live/DOMAIN/cert.pem
/etc/letsencrypt/live/DOMAIN/cert.pem: OK


[root at certs ~]# openssl verify -CAfile 
/etc/letsencrypt/live/DOMAIN/chain.pem 
/etc/letsencrypt/live/DOMAIN/cert.pem
C = US, O = Internet Security Research Group, CN = ISRG Root X1
error 2 at 2 depth lookup: unable to get issuer certificate
error /etc/letsencrypt/live/DOMAIN/cert.pem: verification failed

I suspect this is because the contents of /etc/ssl/certs is so 
different.  On Ubuntu, it contains individual symlinks for all the root 
certs, but on Alma, it just contains ca-bundle.crt, ca-bundle.trust.crt, 
and postfix.pem files.  The root cert is contained in ca-bundle.crt, but 
maybe openssl is not using that.

I fully expect the files downloaded by certbot to verify, so I am not 
checking them directly.  My aim is to make sure that the file I build 
passes verification, just in case my script built it wrong, which could 
happen if certbot's output changes in some way.  Too bad they won't let 
me use Ubuntu.

I experimented and came up with another command that does work:

openssl verify -untrusted le_root.pem -untrusted 
/etc/ssl/certs/local/DOMAIN.wildcards.pem 
/etc/ssl/certs/local/DOMAIN.wildcards.pem

The file named le_root.pem contains JUST the root certificate. Since all 
of the certs generated by this setup will come from LetsEncrypt, I can 
put the root cert in a static file and not worry about changing it until 
they move to a new root.

Thanks for pointing me in the right direction!

Shawn