[openssl-project] DRBGs, threads and locking

[Dr. Matthias St. Pierre]

It is good that Tim hit the break and requested a discussion. That was
overdue and it is unfortunate that we did not start it much earlier. I
think Tim brought up to important points:

1. We need to to pause for a discussion to determine the direction to
go. Otherwise the DRBG implementation will become an ever moving target

2. Instead of guessing about the performance impacts of our change we
should more rely on measuring them.

Ad 1: Unfortunately, it was not clear until a few days ago that there
was so much disagreement on how to do it right. In view of the upcoming
beta freeze, it would probably be the best to leave the status-quo on
master for 1.1.1, and continue the discussion under the premise that it
will be implemented in 1.1.2-dev or 1.2.0-dev. This would give us more
time to think about the optimal solution.

Ad 2: All the haste and the last-minute changes were in some way caused
by the fact that we did not notice the performance regressions until
#5559, because we were not measuring them on a regular base. Having
'openssl speed' is not sufficient and it would really be great if we
could have feedback about the actual performance of "real world" high
performance web servers. But that is out of reach for ordinary persons
and can only be done by a larger company.


per-thread vs. per-ssl

As for the discussion about whether per-thread or per-ssl DRBGs are
better from a security perspective: I am not a professional
cryptographer, so I'm not in the position to decide that. But I can say
that currently the per-thread implementation is far superior when it
comes to simplicity of design. And simplicity of design is an important
countermeasure to prevent bugs and security holes. One of the reasons
why Kurt reverted the per-ssl implementation was that it was a bit ugly
and unsatisfactory, because a lot of changes had to be made to functions
for handing down the correct DRBG through the callstack down to the low
level functions that needed to use it.


If per-ssl DRBGs are desired, I propose the following solution which
reconciles the two approaches without loosing simplicity: If the public
and private DRBGs are thread-local anyway, then it is easy to implement
them as a stack so that the per-thread DRBG can be exchanged by the
per-ssl DRBG within the scope of a function. The correct DRBG would then
be picked up automatically further down in the stack when RAND_bytes()
resp. RAND_priv_bytes() is called.

I am thinking of an API like

    RAND_DRBG_push_public(RAND_DRBG *public);
    RAND_DRBG_push_private(RAND_DRBG *private);

    RAND_DRBG_pop_public();
    RAND_DRBG_push_private();

As said before, this does not imply a preference in one of the two
directions, it's only a suggestion about how it could be implemented.
And I will not throw in a quick pull request for this... ;-)

Matthias