[openssl-dev] [openssl.org #3626] Entropy on Win discounts CryptGenRandom length

[Glen Miner via RT]

rand_win.c(361) RAND_add(buf, sizeof(buf), 0);This is inconsistent with line 375 which passes sizeof(buf) for the bytes of entropy.
This means that the entropy from the OS pool is discounted; in normal circumstances this is insignificant because the rest of this function collects plenty of entropy from the rest of the system.
In rare cases this leads to problems in md_rand.c -- line 403
	ok = (entropy >= ENTROPY_NEEDED);	if (!ok)
In the case that rand_win.c doesn't seed with enough entropy it goes on to stir the pool it but never sets 'ok' afterward and so ssleay_rand_bytes returns failure which seems a bit harsh.
Locally we've changed rand_win.c to pass sizeof(buf) when adding the entropy from PROV_RSA_FULL which avoids the problems in md_rand.c -- I'm not sure what the intention is there, though.
The rare case involved actually came about after we discovered that RAND_poll was locking heaps in our process and generally causing massive hitches in certain parts of our game; once we'd traced it back to this heap-crawling stuff we made two changes to RAND_poll:
1) We increased the size of buf to 10242) We changed line 45 to if (kernel && !good)
The idea is that if the OS entropy pool will feed us a good chunk of good random numbers we can avoid beating the hell out of the heap which required serialization. Initially we found this worked great until certain users started complaining about OpenSSL errors because of the bugs described above -- it turns out some of them had turned off the OS Workstation service which was contributing entropy in rand_win.c(269) which brought us below the required amount of entropy in md_rand.c and here we are.
-g