Cert hot-reloading

[Viktor Dukhovni]

On Tue, Sep 01, 2020 at 11:09:18PM -0400, Michael Richardson wrote:

> My reading of the CABForum push towards 13 month validity, and the
> LetsEncrypt 90-day process it that private key replacements are not
> necessarily replaced that often.

Perhaps so, but the mechanism needs to be general enough to avoid
making that assumption.

> Viktor suggests using the combined private-key/certificate format.
> I think that's a undesired constrait.
> For the paranoid who want to encrypt their private keys and type passphrases
> when whey reload would probably not like that.

The encryption of private keys is at the PEM object level, not at the
file level, so encrypted keys coëxist just fine with cleartext cert
chains in the same file.

> It probably also fails if the private key is in an HSM (and you don't
> replace that private key as often, I imagine).

This is a better argument for also offering a multifile model, because
indeed in that case the key is not even in a file, all one has is some
sort of key handle.  I'm much less familiar with how one handles key
and cert rotation with HSMs.  This used to mean engines, but all that
is changing in OpenSSL 3.0.0, so TBD.  If both the key and the cert
chain became just handles to data fetched each time they're needed,
then the entire problem might become moot.  But it may prove tricky
to present a uniform interface to both PEM files and HSMs that works
robustly for both under their respective operational models.

> In all algorithms I'm aware of, the public key can be derived from the private key,
> so we can check if the loaded private key matches the public key in the
> certificate.  (I seem to remember some attack on some systems where doing that
> checked defeated the attack, but I can't recall which one)

Yes, OpenSSL generally supports checking that the public key in
the leaf certificate matches the public key portion of the private
key, but in fact with HSMs where the public and private keys may
be just opaque handles, such a check is not always possible, in
that case one might actually need to attempt to sign something,
and see whether the signature is valid...  I don't recall the
details.  See 

    https://github.com/openssl/openssl/blob/master/ssl/ssl_rsa.c#L148-L192

and related functions and also:

    https://github.com/openssl/openssl/blob/master/ssl/ssl_rsa.c#L148-L192

> So it seems that we can easily use the mis-match of the keys to trigger a
> check on the private key file, and if we can't load it (passphrase, etc),
> then we could actually reject reloading the certificate file.

Right, and then backoff, retry, etc.  Folks who elect a single-file
format, and execute rollovers correctly can avoid the associated costs.

> Oh, and while I think that openssl should have some reference code that uses
> openat(), I rather suggest that this is reference code.  Let the application
> deal with setting up and processing the file update events, and calling
> OpenSSL to potentially load a new certificate/key pair.
> OpenSSL should focus on the reference counting needed underneath.

Yes, that's fine, but in that case more explicit code is needed at the
application layer to "make it go".  Lots of tradeoffs.

In any case this is a good project for someone suitably skilled.
Ideally, design and documentation before code!

-- 
    Viktor.