Cert hot-reloading

[Kyle Hamilton]

I'm not sure I can follow the "in all cases it's important to keep the key
and cert in the same file" argument, particularly in line with openat()
usage on the cert file after privilege to open the key file has been
dropped.  I agree that key/cert staleness is important to address in some
manner, but I don't think it's necessarily appropriate here.

I also don't think it's necessarily okay to add a new requirement that e.g.
letsencrypt clients reconcatentate their keys and certs, and that all of
the Apache-style configuration guides be rewritten to consolidate the key
and cert files. On a simple certificate renewal without a rekey, the best
current practice is sufficient.  (As well, a letsencrypt client would
possibly need to run privileged in that scenario to reread the private key
file in order to reconcatenate it, which is not currently actually
necessary.  Increasing the privileges required for any non-OS service for
any purpose that isn't related to OS kernel privilege requirements feels a
bit disingenuous.)

Of course, if you want to alter the conditions which led to the best
current practice (and impose retraining on everyone), that's a different
matter.  But I still think increasing privilege requirements would be a bad
thing, under the least-privilege principle.

-Kyle H

On Sun, Aug 30, 2020, 18:36 Viktor Dukhovni <openssl-users at dukhovni.org>
wrote:

> On Sun, Aug 30, 2020 at 05:45:41PM -0500, David Arnold wrote:
>
> > If you prefer this mailing list over github issues, I still want to ask
> > for comments on:
> >
> > Certificate hot-reloading #12753
> > <https://github.com/openssl/openssl/issues/12753>
> >
> > Specifically, my impression is that this topic has died down a bit and
> > from the linked mailing list threads, in my eye, no concrete conclusion
> > was drawn.
> >
> > I'm not sure how to rank this motion in the context of OpenSSL
> > development, but I guess OpenSSL is used to producing ripple effects,
> > so the man-hour argument might be a genuinely valid one.
> >
> > Please inform my research about this issue with your comments!
>
> This is a worthwhile topic.  It has a few interesting aspects:
>
>     1.  Automatic key+cert reloads upon updates of key+cert chain PEM
>         files.  This can be tricky when processes start privileged,
>         load the certs and then drop privs, and are no longer able
>         to reopen the key + cert chain file.
>
>         - Here, for POSIX systems I'd go with an approach where
>           it is the containing directory that is restricted to
>           root or similar, and the actual cert files are group
>           and or world readable.  The process can then keep
>           the directory file descriptor open, and then openat(2)
>           to periodically check the cert file, reloading when
>           the metadata changes.
>
>         - With non-POSIX systems, or applications that don't
>           drop privs, the openat(2) is not needed, and one
>           just checks the cert chain periodically.
>
>         - Another option is to use passphrase-protected keys,
>           and load the secret passphrase at process start from
>           a separate read-protected file, while the actual
>           private key + cert chain file is world readable,
>           with the access control via protecting the passphrase
>           file.
>
>         - In all cases, it is important to keep both the private
>           key and the cert in the same file, and open it just
>           once to read both, avoiding races in which the key
>           and cert are read in a way that results in one or
>           the other being stale.
>
>     2.  Having somehow obtained a new key + cert chain, one
>         now wants to non-disruptively apply them to running
>         servers.  Here there are two potential approaches:
>
>         - Hot plug a new pointer into an existing SSL_CTX structure.
>           While the update itself could be made atomic, the readers
>           of such pointers might read them more than once to separately
>           extract the key and the cert chain, without checking that
>           they're using the same pointer for both operations.
>
>           This is bound to be fragile, though not necessarily
>           impossible.
>
>         - Build a new SSL_CTX, and use it to accept *new* connections,
>           while existing connections use whatever SSL_CTX they started
>           with.  I believe this can work well, because "SSL" handles
>           increment the reference count of the associated SSL_CTX
>           when they're created, and decrement it when destroyed.
>
>           So when you create a replacement SSL_CTX, you can just
>           SSL_CTX_free() the old, and it will only actually
>           be deleted when the last SSL connection tied to that
>           SSL_CTX is destroyed.
>
>           It is true that typical SSL_CTX construction is modestly
>           expensive (loading CA stores and the like) but some of
>           that could be handled by sharing and reference-counting
>           the stores.
>
> So my preferred approach would be to create a new SSL_CTX, and get new
> connections using that.  Now in a multi-threaded server, it could be a
> bit tricky to ensure that the SSL_CTX_free() does not happen before all
> threads reading the pointer to the latest SSL_CTX see the new pointer
> installed.  Something equivalent to RCU may be needed to ensure that the
> free only happens after the new pointer is visible in all threads.
>
> Designs addressing various parts of this would be cool, provided they're
> well thought out, and not just single-use-case quick hacks.
>
> --
>     Viktor.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mta.openssl.org/pipermail/openssl-users/attachments/20200830/ffe5fe29/attachment.html>