Questions about the semantics of SSL_CTX_set_cert_verify_callback

[Viktor Dukhovni]

On Fri, Jul 07, 2023 at 11:32:46AM +0000, Andrew Lee-Thorp wrote:

> I am looking at some code that does this:
> 
> SSL_CTX_set_verify(ctx, SSL_VERIFY_PEER, NULL);

This asks OpenSSL to terminate the handshake when peer certificate
verification fails.

> SSL_CTX_set_cert_verify_callback(ctx, callback, NULL);

This sets up an application-specific certificate *chain* verification
method, overriding the default use of X509_verify_cert(3).  The
specified callback is responsible for all aspects of certificate chain
validation:

    * Name checks
    * Name constraint checks
    * Validity date checks
    * Signature checks
    * Security level checks
    * Path length constraint checks
    * Extended Key Usage checks
    * Possibly process DANE TLSA trust anchors instead
      of the usual CAfile/CApath trust store.
    * ...

Most applications that set up this type of callback just do some
pre/post processing and delegate the real work to an explicit call to
X509_verify_cert(3).

> Elsewhere:
> 
> int callback (X509_STORE_CTX *ctx, void *arg)
> {
> ..
> <elided code>
> 
>     returnValue = X509_STORE_CTX_get0_cert(ctx); <-- ??
>     returnValue2 = X509_STORE_CTX_get0_chain(ctx); <-- ??

The <elided code> is perhaps important here...

> My question is about the callback and the semantics of get0_cert
> and get0_chain in the context of the callback.
> 
> The documentation implies this callback is called once and get0_chain
> returns a constructed and validated chain but I would like to make
> sure.

The EE (end-entity) certificate object is owned by the caller of the
verification callback.  The constructed chain is owned by the
X509_STORE_CTX object, and will be freed when that is freed.

> So, firstly is callback invoked once or multiple times for each cert
> in the chain? I believe its' once but would like to make sure.

It is not possible to "verify" a certificate chain just by looking at
each certificate in isolation.  This callback's responsibility is the
same as that of X509_verify_cert(3), namely to validate the supplied
EE certificate with the aid of any supplied intermediate ("untrusted")
certificates, and the trust store (or DANE TLSA records).

It is called once per chain (EE cert + untrusted issuers).

> Could the callback also be invoked for an unvalidated chain? and if so
> would get0_cert and get0_chain return NULL in such a situation or does
> this not apply?

It is always/only called for an unvalidated chain.  Validating the chain
is the callback's job.  The certificate will likely be the EE
certificate that was supposed to be validated.  The chain will be in
whatever state the elided code left it in (initially NULL).

> If it can be invoked for an unvalidated chain should the code also
> call X509_STORE_CTX_get_error() to get the validation status of the
> chain verification or does this not apply.

The question makes no sense, the chain is always unvalidated, it is the
callback's job to do the validation.

> Does get0_cert return the leaf cert (in the chain) or as the
> documentation suggest the current cert being processed by the ctx?

Leaf, there is no documentation suggesting otherwise.  Perhaps
you were looking at X509_STORE_CTX_get_current_cert().

-- 
    Viktor.