Cert hot-reloading

[Karl Denninger]

On 8/30/2020 19:28, Viktor Dukhovni wrote:
> On Sun, Aug 30, 2020 at 05:45:41PM -0500, David Arnold wrote:
>
>> If you prefer this mailing list over github issues, I still want to ask
>> for comments on:
>>
>> Certificate hot-reloading #12753
>> <https://github.com/openssl/openssl/issues/12753>
>>
>> Specifically, my impression is that this topic has died down a bit and
>> from the linked mailing list threads, in my eye, no concrete conclusion
>> was drawn.
>>
>> I'm not sure how to rank this motion in the context of OpenSSL
>> development, but I guess OpenSSL is used to producing ripple effects,
>> so the man-hour argument might be a genuinely valid one.
>>
>> Please inform my research about this issue with your comments!
> This is a worthwhile topic.  It has a few interesting aspects:
>
>      1.  Automatic key+cert reloads upon updates of key+cert chain PEM
>          files.  This can be tricky when processes start privileged,
>          load the certs and then drop privs, and are no longer able
>          to reopen the key + cert chain file.
>
>          - Here, for POSIX systems I'd go with an approach where
>            it is the containing directory that is restricted to
>            root or similar, and the actual cert files are group
>            and or world readable.  The process can then keep
>            the directory file descriptor open, and then openat(2)
>            to periodically check the cert file, reloading when
>            the metadata changes.
>
>          - With non-POSIX systems, or applications that don't
>            drop privs, the openat(2) is not needed, and one
>            just checks the cert chain periodically.
>
>          - Another option is to use passphrase-protected keys,
>            and load the secret passphrase at process start from
>            a separate read-protected file, while the actual
>            private key + cert chain file is world readable,
>            with the access control via protecting the passphrase
>            file.
>
>          - In all cases, it is important to keep both the private
>            key and the cert in the same file, and open it just
>            once to read both, avoiding races in which the key
>            and cert are read in a way that results in one or
>            the other being stale.
>
>      2.  Having somehow obtained a new key + cert chain, one
>          now wants to non-disruptively apply them to running
>          servers.  Here there are two potential approaches:
>
>          - Hot plug a new pointer into an existing SSL_CTX structure.
>            While the update itself could be made atomic, the readers
>            of such pointers might read them more than once to separately
>            extract the key and the cert chain, without checking that
>            they're using the same pointer for both operations.
>
>            This is bound to be fragile, though not necessarily
>            impossible.
>
>          - Build a new SSL_CTX, and use it to accept *new* connections,
>            while existing connections use whatever SSL_CTX they started
>            with.  I believe this can work well, because "SSL" handles
>            increment the reference count of the associated SSL_CTX
>            when they're created, and decrement it when destroyed.
>
>            So when you create a replacement SSL_CTX, you can just
>            SSL_CTX_free() the old, and it will only actually
>            be deleted when the last SSL connection tied to that
>            SSL_CTX is destroyed.
>
>            It is true that typical SSL_CTX construction is modestly
>            expensive (loading CA stores and the like) but some of
>            that could be handled by sharing and reference-counting
>            the stores.
>
> So my preferred approach would be to create a new SSL_CTX, and get new
> connections using that.  Now in a multi-threaded server, it could be a
> bit tricky to ensure that the SSL_CTX_free() does not happen before all
> threads reading the pointer to the latest SSL_CTX see the new pointer
> installed.  Something equivalent to RCU may be needed to ensure that the
> free only happens after the new pointer is visible in all threads.
>
> Designs addressing various parts of this would be cool, provided they're
> well thought out, and not just single-use-case quick hacks.

This works now; I use it with an application that checks in with a 
license server and can grab a new cert.  OpenSSL appears to have no 
problem with setting up a new SSL_CTX and using it for new connections; 
the old ones continue onward until they terminate, and new ones are fine 
as well.

This appears to be be ok with the current code; I've yet to have it blow 
up in my face although at present the certs in question are reasonably 
long-lived.  Whether it's robust enough to handle very short-term 
certificates I do not know.

-- 
Karl Denninger
karl at denninger.net <mailto:karl at denninger.net>
/The Market Ticker/
/[S/MIME encrypted email preferred]/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mta.openssl.org/pipermail/openssl-users/attachments/20200830/dc260647/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4897 bytes
Desc: S/MIME Cryptographic Signature
URL: <https://mta.openssl.org/pipermail/openssl-users/attachments/20200830/dc260647/attachment.bin>