Query on SSL Mutual Authentication on Server

[Viktor Dukhovni]

On Mon, Mar 01, 2021 at 09:21:29PM +0530, Archana wrote:

> I am new to SSL programming. On our SSL Server implementation, we are
> trying to enforce Mutual Authentication. Is it Mandatory to provide a user
> defined Callback using SSL_ctx_setverify()

No callback is required (callbacks are primarily useful for logging,
though they can also, with care, be used to make chain verification
more "permissive", but there be dragons).  However, you must then
still call:

    int mode = SSL_VERIFY_PEER
             | SSL_VERIFY_FAIL_IF_NO_PEER_CERT
             | SSL_VERIFY_CLIENT_ONCE;

    SSL_CTX_set_verify(ctx, mode, NULL);

to set the verification mode to request (and enforce) the presence of a
client certificate.  Depending on the client, you may also need to make
sure to provide a non-empty list of client CA hints that includes all
the trust-anchor CAs from which you'll accept client certificate chains.
(Clients using Java SSL APIs typically require that to be the case).

This can be done via:

    const char *CAfile = "/your/CA/file";
    STACK_OF(X509_NAME) *calist = SSL_load_client_CA_file(CAfile);

    if (calist == NULL) {
        /* log error loading client CA names */
    }
    SSL_CTX_set_client_CA_list(server_ctx, calist);

> If yes, Is it expected to do the IP or hostname validation?

Neither, authorization of the client is up to you.  OpenSSL will check
the dates, validity of the signatures, ... in the clients certificate
chain, but checking whether any of the subject names in the client
certificate are allowed to access your server is up to you.

There is no prior expectation that the client's certificate is
specifically related to its IP address or hostname.

You may in fact, depending on the structure of your code, be able to
configure the expected client name prior to the SSL handshake
with SSL_accept(3), but after accepting the client TCP connection.

To set the expected hostname(s), see the documentation of:

    int SSL_set1_host(SSL *s, const char *hostname);
    int SSL_add1_host(SSL *s, const char *hostname);

For IP addresses, there's a slightly lower-level interface:

        X509_VERIFY_PARAM *SSL_CTX_get0_param(SSL_CTX *ctx);

        int X509_VERIFY_PARAM_set1_ip(X509_VERIFY_PARAM *param,
                                      const unsigned char *ip, size_t iplen);
        int X509_VERIFY_PARAM_set1_ip_asc(X509_VERIFY_PARAM *param, const char *ipasc);

or after the handshake completes, you can call one of:

        int X509_check_host(X509 *, const char *name, size_t namelen,
                            unsigned int flags, char **peername);
        int X509_check_email(X509 *, const char *address, size_t addresslen,
                             unsigned int flags);
        int X509_check_ip(X509 *, const unsigned char *address, size_t addresslen,
                          unsigned int flags);
        int X509_check_ip_asc(X509 *, const char *address, unsigned int flags);

-- 
    Viktor.