Cert hot-reloading

[Viktor Dukhovni]

On Sun, Aug 30, 2020 at 05:45:41PM -0500, David Arnold wrote:

> If you prefer this mailing list over github issues, I still want to ask 
> for comments on:
> 
> Certificate hot-reloading #12753
> <https://github.com/openssl/openssl/issues/12753>
> 
> Specifically, my impression is that this topic has died down a bit and 
> from the linked mailing list threads, in my eye, no concrete conclusion 
> was drawn.
> 
> I'm not sure how to rank this motion in the context of OpenSSL
> development, but I guess OpenSSL is used to producing ripple effects,
> so the man-hour argument might be a genuinely valid one.
> 
> Please inform my research about this issue with your comments!

This is a worthwhile topic.  It has a few interesting aspects:

    1.  Automatic key+cert reloads upon updates of key+cert chain PEM
        files.  This can be tricky when processes start privileged,
        load the certs and then drop privs, and are no longer able
        to reopen the key + cert chain file.

        - Here, for POSIX systems I'd go with an approach where
          it is the containing directory that is restricted to
          root or similar, and the actual cert files are group
          and or world readable.  The process can then keep
          the directory file descriptor open, and then openat(2)
          to periodically check the cert file, reloading when
          the metadata changes.

        - With non-POSIX systems, or applications that don't
          drop privs, the openat(2) is not needed, and one
          just checks the cert chain periodically.

        - Another option is to use passphrase-protected keys,
          and load the secret passphrase at process start from
          a separate read-protected file, while the actual
          private key + cert chain file is world readable,
          with the access control via protecting the passphrase
          file.

        - In all cases, it is important to keep both the private
          key and the cert in the same file, and open it just
          once to read both, avoiding races in which the key
          and cert are read in a way that results in one or
          the other being stale.

    2.  Having somehow obtained a new key + cert chain, one
        now wants to non-disruptively apply them to running
        servers.  Here there are two potential approaches:

        - Hot plug a new pointer into an existing SSL_CTX structure.
          While the update itself could be made atomic, the readers
          of such pointers might read them more than once to separately
          extract the key and the cert chain, without checking that
          they're using the same pointer for both operations.

          This is bound to be fragile, though not necessarily
          impossible.

        - Build a new SSL_CTX, and use it to accept *new* connections,
          while existing connections use whatever SSL_CTX they started
          with.  I believe this can work well, because "SSL" handles
          increment the reference count of the associated SSL_CTX
          when they're created, and decrement it when destroyed.

          So when you create a replacement SSL_CTX, you can just
          SSL_CTX_free() the old, and it will only actually
          be deleted when the last SSL connection tied to that
          SSL_CTX is destroyed.

          It is true that typical SSL_CTX construction is modestly
          expensive (loading CA stores and the like) but some of
          that could be handled by sharing and reference-counting
          the stores.

So my preferred approach would be to create a new SSL_CTX, and get new
connections using that.  Now in a multi-threaded server, it could be a
bit tricky to ensure that the SSL_CTX_free() does not happen before all
threads reading the pointer to the latest SSL_CTX see the new pointer
installed.  Something equivalent to RCU may be needed to ensure that the
free only happens after the new pointer is visible in all threads.

Designs addressing various parts of this would be cool, provided they're
well thought out, and not just single-use-case quick hacks.

-- 
    Viktor.