[openssl-users] RNG behavior by default

Mon Jan 7 18:31:36 UTC 2019

A wonderful Monday in the beautiful Winter time i wish.

I am sorry for the late reply again, i got a bug report for the
mailer i maintain, and from a long time user.
I hope it is ok that i compress the answers in one message, i am
talking much too much...

Kurt Roeckx wrote in <20190105221506.GA18502 at roeckx.be>:
 |On Sat, Jan 05, 2019 at 08:33:18PM +0100, Steffen Nurpmeso wrote:
 |> 
 |> (I am also really interested and will look into OpenSSL to see if
 |> the abort() that seems to happen if the initial seed fails is in
 |> a linker-resolved constructor, and if not, why later failures do
 |> not also abort.
 |
 |We do not call abort(). A function like RAND_bytes() or
 |RAND_status() will just return an error.

Aah, i see.  I obviously had a completely false impression of the
new design.  I do not know where this came from, if i now search
in doc/ of the [master] there no mention of such wording, neither
abort nor exit nor whatever.  Hmm.?

  ...
 |RAND_bytes() has always documented that it can fail. Most function
 |calls can return errors for various reasons. Not checking for such
 |errors is in my opinion a bad style.

Well i, and until just now, interpreted the PRNG as something like
a "cipher stream", which simply applies (magic, to me) math to
a buffer of undefined content and an algorithm-chosen size.  Just
like MD5, which' RFC (1321) does not have error conditions.  (A
digest, but i had the RFC number at hand.)

So, to me.., i do not see any possible error condition, since the
initial seeding has been testified with RAND_status().

This is different now, and i will change the implementation as
soon as possible.  (This week.)

 |> i do not like that the stuff is instantiated in all threads
 |
 |It's only created in each thread that tries to the RNG. I'm not
 |sure what exactly your problem with it is. Either we need a global
 |lock, or we need an RNG per thread. We went with the per thread
 |RNG.

All my fault.  As a side note, i remember a commit message of
DragonFlyBSD and M. Dillon fly by (some years ago) where he
utilized the effect of SMP races as a random source.  If i recall
correctly.  Of course you had to protect the reseed count or
whatever (but there atomic cas or a spinlock would suffice
i guess).

 |We have a master DRBG that you can get with
 |RAND_DRBG_get0_master(). I recommend that you do not use it. It
 |requires that you take an external lock to use it. Internally we
 |lock it, but there is no way for you to use the same lock.
 |
 |> which
 |> in addition requires forks handlers to be installed all over the
 |> place.
 |
 |OpenSSL adds it's own fork handler now. You should not need to do
 |anything on fork related to OpenSSL.

It is just, to me, lack of standard.  But i do not dlopen() you,
and i do not dlclose() you.  In this scenario libraries may do use
anything, install atexit, onfork, whatever.  Other SSL libraries
even removed the possibility to hook the memory allocation
methods!

 |> the Linux kernel that drives
 |> the world from smallest to hugest has one internal entropy pool
 |> that feeds two public pools, whereas i the lucent little hobby
 |> server from user space get an armada of these.  Wow.
 |
 |Linux has a master pool, and a pool per core for the very same
 |reason as we also have a master DRBG and per thread DRBG.

I have read [1], but not that thorougly.  I was referring to the
user space interface, anyway.  (The thing is, i would have
preferred reading a one or two pages abstract instead of flying
over 138 pages.  I seem to know he knows.  I have read some
dissertations in my life, and i like the dedication if people
translate a complete system manual to German there, or show in
detail that they know know know their topic.  Having said that,
a very nice such work i have read from the Uni Kaiserslautern, it
was about 30 pages and it had been clear that "he can", shaking
off his hand.  Why need more?  Very good.  Imho.)

  [1] https://www.bsi.bund.de/SharedDocs/Downloads/EN/BSI/Publications/Studies/LinuxRNG/LinuxRNG_EN.pdf?__blob=publicationFile&v=7

 --End of <20190105221506.GA18502 at roeckx.be>

Thanks.  I did just now.

 |> That is really bad.  Of course you had to do it like this, and you
 |> surely have looked around to see what servers and other software
 |> which use OpenSSL do with the PRNG after forking (i.e., whether
 |> they reiterate the [RAND_file_name(),] RAND_load_file(),
 |> [:[RAND_add(),] RAND_status()], [RAND_write_file()] dance as
 |> documented, or not).
 |
 |I really don't understand your frustration: the new PRNG was designed
 |to relieve the application from the burden of seeding. It is easier to use,
 |not more complicated: No need to call RAND_add(), RAND_seed(),
 |RAND_load_file() and all this stuff. Just make sure the os entropy sources
 |are available and then simply use RAND_bytes(). But don't expect it to
 |succeed always.

And may i ask, what to do if it fails?  Try it in a loop a finite
number of times, assuming it over-and-over detects it is in an
error state and tries to reseed unless that succeeds, and bail if
still without success thereafter?

Talking about the only program in the public that i represent:
i will not abort that one just because a PRNG fails to produce
some bytes that this program uses for the purpose of, well, having
some (pseudo but) random bytes; because: this is _completely_
disproportionate to the use cases of those random numbers.
I guess the same could be said about UUIDS generated via PRNGs
instead of how the standard defines them, etc. etc.
Please do not say "use random(3) or rand(3)" now, please.
So what to do?

 |> I think i will move away from RAND_ then, nonetheless, and at
 |> least for the things i have control of.
 |
 |I don't think this is a good idea. In the case when there is no suitable \
 |entropy
 |source, the OpenSSL RNG will fail to generate random bytes, indeed.
 |But how do you expect your application to seed the RNG properly in this
 |case? What is your application's entropy source? If you have one, which

Yes, that is the problem.  We do have several possibilities for
VAL_RANDOM.  Some of them only seed our builtin ARC4
implementation, which until now exposes the internal buffer as
such (sufficient for our purpose yet).  Examples of this are
/dev/urandom, Linux getrandom as system call or library call,
aehh...  If that seeding fails, we create a completely
unscientific homebrew seed via CLOCK_REALTIME or gettimeofday(2),
whatever is available, wildly XORing through the buffer, further
randomizing those weakly through the well-known algorithm from
"Random number generators: good ones are hard to find", Park and
Miller, Communications of the ACM, vol. 31, no. 10, October 1988,
p. 1195 (a_aux_rand_weak()).  This is the code:

   if(n_poption & n_PO_D_V)
      n_err(_("P(seudo)R(andomNumber)G(enerator): creating homebrew seed\n"));
   for(seed = (uintptr_t)a_aux_rand & UI32_MAX, rnd = 21; rnd != 0; --rnd){
      for(u.i = n_NELEM(a_aux_rand->b32); u.i-- != 0;){
         ui32_t t, k;

# ifdef mx_HAVE_CLOCK_GETTIME
         clock_gettime(CLOCK_REALTIME, &ts);
         t = (ui32_t)ts.tv_nsec;
# else
         gettimeofday(&ts, NULL);
         t = (ui32_t)ts.tv_usec;
# endif
         if(rnd & 1)
            t = (t >> 16) | (t << 16);
         a_aux_rand->b32[u.i] ^= a_aux_rand_weak(seed ^ t);
         a_aux_rand->b32[t % n_NELEM(a_aux_rand->b32)] ^= seed;
         if(rnd == 7 || rnd == 17)
            a_aux_rand->b32[u.i] ^= a_aux_rand_weak(seed ^ (ui32_t)ts.tv_sec);
         k = a_aux_rand->b32[u.i] % n_NELEM(a_aux_rand->b32);
         a_aux_rand->b32[k] ^= a_aux_rand->b32[u.i];
         seed ^= a_aux_rand_weak(a_aux_rand->b32[k]);
         if((rnd & 3) == 3)
            seed ^= su_prime_lookup_next(seed);
      }

So this is totally unscientific nonsense and overkill in one, but
at least we mix the low bits into the upper ones of the smallest
moving time fractions, and as such perform better than many
C libraries seem to have done in the past, as far as i know.
I think this is the best i can do.  Wait.  We _could_ check
whether we have subsecond precision sleep, and do nanosleep(0) do
gain some kind of sched_yield().

Others completely replace all this code: if we find a native
arc4random(3) (posix_random(3) will not become true,
unfortunately) we use that.  If we are linked against OpenSSL, we
do use yours exclusively -- but this possibly has to be changed.

 |OpenSSL does not handle yet, you could theoretically register your
 |own get_entropy callback for the master DRBG at application startup time.

So by hooking in such after the initial seeding succeeded (as
testified via RAND_status()) i can ensure that further reseeding
goes over my table and will (thus) always succeed?  I will look
into this possibility, thanks for the suggestion!

Well, as above, it is disproportionate to the use case of randoms
here.  (But this ARC4 stuff is a port of an old C++ library class,
and will move to an external C library in the future.  That is to
say, for now we could get by simply with a_aux_rand_weak(), but
since this code is old and what i have used so long.)

And i for one think that "snake oil security" is a wording much
too strong in the case of reseeding of a very secure stream
cipher, the entropy pool of which is protected by a very secure
distorting digest.  I do not know about forward security, however.
And i have read in the documented quoted above however, that the
in-between-boots random entropy file is not even counted against
security no more, which is why sshd blocks booting a minute.
That i find total overkill.

 |P.S: As for automatic reseeding: There are potential issues when upgrading
 |applications which do a fork and chroot from 1.1.0 to 1.1.1. Because in
 |version 1.1.1, the application will reseed after fork, and the reseeding \
 |can fail
 |if the application developer (or administrator) did not pay attention \
 |to provide
 |a random device in the chroot jail. This was no problem with 1.1.0 \
 |and below,
 |because the RNG never reseeded automatically.
 |
 |OpenSSL does everything to avoid such a chroot reseeding failure (#6432 \
 |and #7437).
 |Also, on modern linux systems OpenSSL will prefer the getrandom system call
 |which does not have this limitation of the random devices. 
 |
 |https://github.com/openssl/openssl/pull/6432
 |https://github.com/openssl/openssl/pull/7437

Thanks for the information.  (It does not apply here, but thank
you nonetheless!)

 --End of <bb6e16e3fee246e887f185d3fbbfd3a2 at Ex13.ncp.local>

Ciao from Germany,

--steffen
|
|Der Kragenbaer,                The moon bear,
|der holt sich munter           he cheerfully and one by one
|einen nach dem anderen runter  wa.ks himself off
|(By Robert Gernhardt)