Strange problem: openssl verify not working on Proxmox VM, works on a bare metal system

[Viktor Dukhovni]

On Sat, Sep 03, 2022 at 05:47:25PM -0600, Shawn Heisey via openssl-users wrote:

> > Post the output of:
> >
> > $ openssl crl2pkcs7 -nocrl -certfile /etc/ssl/certs/local/DOMAIN.wildcards.pem |
> >     openssl pkcs7 -print_certs -noout |
> >     perl -ne 'BEGIN{$/="\n\n\n"} s/\n+/\n/g; print $_, "\n"'
> 
> subject=CN = DOMAIN
> issuer=C = US, O = Let's Encrypt, CN = R3
> 
> subject=C = US, O = Let's Encrypt, CN = R3
> issuer=C = US, O = Internet Security Research Group, CN = ISRG Root X1

Your cert file holds the EE certificate and its R3 intermediate issuer.
It does not include a trusted root, therefore when used as the sole
trust store, it is not sufficient to verify your EE certificate.

If your system's default CApath (also consulted) happens to contain the
"ISRG Root X1" issuer CA of "R3", then the certificate validates,
otherwise it does not.  If that ISRG X1 is only in the default CA file,
verification fails, because you've specified your chain as the CAfile
instead.

> The combined cert file that I have built does NOT contain the root 
> cert.  I only need to send the server cert and the issuing cert. The 
> browser will have the root cert, so there is no need to send it.

This is typically true (unless you've published DANE TLSA records for
the root CA), but to match the browser's behaviour, your trust store
needs to include the ISRG X1 root, and it does not when you override
the CAfile, and CApath fails to bridge the gap.

> If I use a file that DOES contain the root cert (one of the files 
> downloaded by certbot) then I can get it to pass with -untrusted, but 
> not -CAfile:
> 
> [root at certs ~]# openssl verify -untrusted /etc/letsencrypt/live/DOMAIN/chain.pem /etc/letsencrypt/live/DOMAIN/cert.pem
> /etc/letsencrypt/live/DOMAIN/cert.pem: OK

The chain file probably contains the cross-certificate for X1 signed by
DST (Digital Signature Trust), and the latter root is in your default
system-wide CAfile or CApath.

> [root at certs ~]# openssl verify -CAfile /etc/letsencrypt/live/DOMAIN/chain.pem /etc/letsencrypt/live/DOMAIN/cert.pem
> C = US, O = Internet Security Research Group, CN = ISRG Root X1
> error 2 at 2 depth lookup: unable to get issuer certificate
> error /etc/letsencrypt/live/DOMAIN/cert.pem: verification failed

This invocation does not add the cross certificate.

> I suspect this is because the contents of /etc/ssl/certs is so 
> different.

Some already list the ISRG X1 root in the system CAfile or CApath,
others do not.  OpenSSL does not ship any trusted certificates.
Your issue has nothing to do with OpenSSL as such.  It is all about
the content of the vendor provided trust store and the command-line
options you're choosing.

-- 
    Viktor.