Cert hot-reloading

[David Arnold]

Should aspects of an implementation be configurable behavior with a 
sane default? I'd guess so...

Hot-plugging the pointer seems to force atomicity considerations 
down-stream, which might be
educationally a good thing for openssl to press for. It also addresses 
Jordan's use case, for however
application specific it might be. For compat reasons, a "legacy" mode 
which creates a new context
for *new* connections might be the necessary "bridge" into that 
transformation.

For change detection: I think "on next authentication" has enough (or 
even better) guarantees over a periodic loop.

For file read atomicity: What are the options to keep letsencrypt & co 
at comfort? Although the hereditary
"right (expectation) for comfort" is somewhat offset by a huge gain in 
functionality. It still feels like a convincing deal.

- add a staleness check on every change detection? (maybe costly?)
- consume a tar if clients want those guarantees? (opt-out or opt-out?)




On Sun, Aug 30, 2020 at 19:54, Kyle Hamilton <aerowolf at gmail.com> wrote:
> I'm not sure I can follow the "in all cases it's important to keep 
> the key and cert in the same file" argument, particularly in line 
> with openat() usage on the cert file after privilege to open the key 
> file has been dropped.  I agree that key/cert staleness is important 
> to address in some manner, but I don't think it's necessarily 
> appropriate here.
> 
> I also don't think it's necessarily okay to add a new requirement 
> that e.g. letsencrypt clients reconcatentate their keys and certs, 
> and that all of the Apache-style configuration guides be rewritten to 
> consolidate the key and cert files. On a simple certificate renewal 
> without a rekey, the best current practice is sufficient.  (As well, 
> a letsencrypt client would possibly need to run privileged in that 
> scenario to reread the private key file in order to reconcatenate it, 
> which is not currently actually necessary.  Increasing the privileges 
> required for any non-OS service for any purpose that isn't related to 
> OS kernel privilege requirements feels a bit disingenuous.)
> 
> Of course, if you want to alter the conditions which led to the best 
> current practice (and impose retraining on everyone), that's a 
> different matter.  But I still think increasing privilege 
> requirements would be a bad thing, under the least-privilege 
> principle.
> 
> -Kyle H
> 
> On Sun, Aug 30, 2020, 18:36 Viktor Dukhovni 
> <openssl-users at dukhovni.org <mailto:openssl-users at dukhovni.org>> 
> wrote:
>> On Sun, Aug 30, 2020 at 05:45:41PM -0500, David Arnold wrote:
>> 
>>  > If you prefer this mailing list over github issues, I still want 
>> to ask
>>  > for comments on:
>>  >
>>  > Certificate hot-reloading #12753
>>  > <<https://github.com/openssl/openssl/issues/12753>>
>>  >
>>  > Specifically, my impression is that this topic has died down a 
>> bit and
>>  > from the linked mailing list threads, in my eye, no concrete 
>> conclusion
>>  > was drawn.
>>  >
>>  > I'm not sure how to rank this motion in the context of OpenSSL
>>  > development, but I guess OpenSSL is used to producing ripple 
>> effects,
>>  > so the man-hour argument might be a genuinely valid one.
>>  >
>>  > Please inform my research about this issue with your comments!
>> 
>>  This is a worthwhile topic.  It has a few interesting aspects:
>> 
>>      1.  Automatic key+cert reloads upon updates of key+cert chain 
>> PEM
>>          files.  This can be tricky when processes start privileged,
>>          load the certs and then drop privs, and are no longer able
>>          to reopen the key + cert chain file.
>> 
>>          - Here, for POSIX systems I'd go with an approach where
>>            it is the containing directory that is restricted to
>>            root or similar, and the actual cert files are group
>>            and or world readable.  The process can then keep
>>            the directory file descriptor open, and then openat(2)
>>            to periodically check the cert file, reloading when
>>            the metadata changes.
>> 
>>          - With non-POSIX systems, or applications that don't
>>            drop privs, the openat(2) is not needed, and one
>>            just checks the cert chain periodically.
>> 
>>          - Another option is to use passphrase-protected keys,
>>            and load the secret passphrase at process start from
>>            a separate read-protected file, while the actual
>>            private key + cert chain file is world readable,
>>            with the access control via protecting the passphrase
>>            file.
>> 
>>          - In all cases, it is important to keep both the private
>>            key and the cert in the same file, and open it just
>>            once to read both, avoiding races in which the key
>>            and cert are read in a way that results in one or
>>            the other being stale.
>> 
>>      2.  Having somehow obtained a new key + cert chain, one
>>          now wants to non-disruptively apply them to running
>>          servers.  Here there are two potential approaches:
>> 
>>          - Hot plug a new pointer into an existing SSL_CTX structure.
>>            While the update itself could be made atomic, the readers
>>            of such pointers might read them more than once to 
>> separately
>>            extract the key and the cert chain, without checking that
>>            they're using the same pointer for both operations.
>> 
>>            This is bound to be fragile, though not necessarily
>>            impossible.
>> 
>>          - Build a new SSL_CTX, and use it to accept *new* 
>> connections,
>>            while existing connections use whatever SSL_CTX they 
>> started
>>            with.  I believe this can work well, because "SSL" handles
>>            increment the reference count of the associated SSL_CTX
>>            when they're created, and decrement it when destroyed.
>> 
>>            So when you create a replacement SSL_CTX, you can just
>>            SSL_CTX_free() the old, and it will only actually
>>            be deleted when the last SSL connection tied to that
>>            SSL_CTX is destroyed.
>> 
>>            It is true that typical SSL_CTX construction is modestly
>>            expensive (loading CA stores and the like) but some of
>>            that could be handled by sharing and reference-counting
>>            the stores.
>> 
>>  So my preferred approach would be to create a new SSL_CTX, and get 
>> new
>>  connections using that.  Now in a multi-threaded server, it could 
>> be a
>>  bit tricky to ensure that the SSL_CTX_free() does not happen before 
>> all
>>  threads reading the pointer to the latest SSL_CTX see the new 
>> pointer
>>  installed.  Something equivalent to RCU may be needed to ensure 
>> that the
>>  free only happens after the new pointer is visible in all threads.
>> 
>>  Designs addressing various parts of this would be cool, provided 
>> they're
>>  well thought out, and not just single-use-case quick hacks.
>> 
>>  --
>>      Viktor.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mta.openssl.org/pipermail/openssl-users/attachments/20200830/635c3cc5/attachment-0001.html>