[openssl-users] Help with OpenSSL running on OSX

[Jeffrey Walton]

On Thu, Jul 9, 2015 at 2:14 AM, Matthew Donald
<matthew.b.donald at gmail.com> wrote:
> One of Imapfilter's users is having problems verifying certificates.  They
> are running Imapfilter on OSX, which I don't have access to.  In addition, I
> understand that OSX runs a custom version of OpenSSL, which has changes to
> the way certificates are verified.

1.1.0 added hostname verification. It affects all version of OpenSSL,
and is not related to OS X or Macs. Its based on Viktor's work. See
X509_check_host(3) and friends
(https://www.openssl.org/docs/crypto/X509_check_host.html).

OS X itself provides OpenSSL 0.9.8. Its pretty anemic by current yardsticks.

OS X also has twists, like it always links to a shared version of the
library is available (even if -Bstatic is used; and even on iOS, where
dylibs are not allowed). It also uses DYLD_LIBRARY_PATH rather than
LD_LIBRARY_PATH (see
https://developer.apple.com/library/mac/documentation/Darwin/Reference/ManPages/man1/dyld.1.html).
Also, OS X does not honor RPATHs.

> Could someone help me debug the issue by telling me:
>
> 1. Does OSX use a certificates file (like other FreeBSD-style systems) to
> hold trusted/root certificates, or does it use some other mechanism?
> 2.  If it uses a file, what is the file name
> 3. if OSX uses some other mechanism to store trusted/root certs, please give
> a link to documentation on how it works.

Yes and no. OS X natively uses a Keychain. See, for example, these
instructions at http://wiki.cacert.org/FAQ/ImportRootCert#Mac_OS_X.

Just because the OS provides a Keychain, does not mean utilities
actually use it. For example, cURL and Subversion, provided by Apple,
does not use it.

However, OpenSSL has its own mechanisms to use a trust store (re:
c_hash and friends). So it depends on how OpenSSL was configured,
which also depends on how ImapFilter configured things.

And I seem to recall OpenSSL does not use it (the project could
probably use an ENGINE to interface to it).

*****

Because of issues with downlevel versions of the library on nearly all
platforms, I use similar to the following to ensure I get what I
compiled and linked against:

    ostringstream oss;
    long version = SSLeay();

    if (version != OPENSSL_VERSION_NUMBER)
    {
        oss << "expected OpenSSL version " << std::dec <<
OPENSSL_VERSION_NUMBER;
        oss << " (" << std::hex << OPENSSL_VERSION_NUMBER << "), but
got version ";
        oss << std::dec << version << " (" << std::hex << version << ")";
        throw runtime_error(oss.str().c_str());
    }

You can actually relax that a bit by only comparing the high bytes.
But I'm not interested in binary compatibility. I want to ensure I'm
linking to precisely what I expect.

For details on versioning and binary compatibility, see How does the
versioning scheme work?,
https://www.openssl.org/support/faq.html#MISC8.

Jeff