Strange problem: openssl verify not working on Proxmox VM, works on a bare metal system

[Shawn Heisey]

On an AlmaLinux 8.6 VM hosted in Proxmox:

[root at certs ~]# openssl verify -CAfile 
/etc/ssl/certs/local/DOMAIN.wildcards.pem 
/etc/ssl/certs/local/DOMAIN.wildcards.pem
C = US, O = Let's Encrypt, CN = R3
error 2 at 1 depth lookup: unable to get issuer certificate
error /etc/ssl/certs/local/DOMAIN.wildcards.pem: verification failed
[root at certs ~]# openssl version
OpenSSL 1.1.1k  FIPS 25 Mar 2021

The VM has CPUs of type host.  It did the same with a CPU type of 
kvm64.  I switched to host because kvm64 did not pass the aes CPU flag 
through.

If I copy the PEM file to a bare metal system running Ubuntu Server 
20.04, it verifies:

elyograg at bilbo:~$ openssl verify -CAfile DOMAIN.wildcards.pem 
DOMAIN.wildcards.pem
DOMAIN.wildcards.pem: OK
elyograg at bilbo:~$ openssl version
OpenSSL 1.1.1f  31 Mar 2020

Other bare metal systems and their results with the same PEM file:

Verifies on Proxmox (the one running the VM) with openssl 1.1.1n
Verifies on Ubuntu 22.04 with openssl 3.0.2
Fails on CentOS 7.5.1804 with openssl 1.0.2k-fips

----

I also have the quictls fork of openssl on the VM, built from source, 
and it fails with exactly the same error message:

[root at certs ~]# /usr/local/bin/qssl verify -CAfile 
/etc/ssl/certs/local/DOMAIN.wildcards.pem 
/etc/ssl/certs/local/DOMAIN.wildcards.pem
C = US, O = Let's Encrypt, CN = R3
error 2 at 1 depth lookup: unable to get issuer certificate
error /etc/ssl/certs/local/DOMAIN.wildcards.pem: verification failed
[root at certs ~]# /usr/local/bin/qssl version
OpenSSL 3.0.5+quic 5 Jul 2022 (Library: OpenSSL 3.0.5+quic 5 Jul 2022)

---

I have redacted the domain name from the filename in what I pasted 
above, but everything else is untouched.  The PEM file contains the 
server cert, the letsencrypt issuing cert, the private key, and 
generated dhparams.  It works for most software that can handle PEM 
files for TLS.  The only software I am sure about that utilizes the 
dhparams is haproxy.  In case it matters, the server cert has a 4096 bit 
key.  The certbot program is functioning correctly.

Does anyone have any idea why this would fail in this way?  Is there 
some information I can gather that would help with troubleshooting?  The 
little evidence I have says it is failing on RPM distros and passing on 
DEB distros.  But the sample size is way too small to adequately support 
that hypothesis.

In the unlikely event this is an XY problem, here is the X:  I am trying 
to set up a letsencrypt certificate creation/renewal system on the VM 
for work that I have running on my own server.  One of the things that I 
have my script doing is validating the certificate file that it produces 
before it declares success.  I would like the VM to do the same, but 
right now I can't because of this issue.

Thanks,
Shawn