[openssl-users] Using FIPS mode and modifying apps

[Dr. Stephen Henson]

On Wed, Jan 28, 2015, jonetsu at teksavvy.com wrote:

> On Mon, 26 Jan 2015 22:35:12 -0500
> Tom Francis <thomas.francis.jr at pobox.com> wrote:
> 
> Thanks for the detailed comments.  I understand the concerns, although
> there's one thing I do not see clearly, that is:
> 
> > 2) Applications that don???t know they???re operating in FIPS
> > mode may attempt to use algorithms that are disallowed in FIPS mode,
> > but using an API that will actually succeed.  
> 
> How could this happen ?  Do you have a practical use case ?  Wouldn't
> OpenSSL in FIPS mode prevent the use of such algorithm in the first
> place ?
> 

OpenSSL does prevent the use of non-FIPS algorithms in FIPS mode but 
applications don't always cleanly handle it.

If the application uses TLS then it should be OK as non-FIPS ciphersuites
are automatically blocked. A possible gotcha is use of private keys encrypted
using the old MD5 PBE algorithm.

If the application uses algorithms directly via for example EVP_DigestInit_ex()
then things are different. A non-FIPS application might never see an error
from EVP_DigestInit_ex() and not handle it properly (e.g. ignoring the
return value). This will cause a hard failure later on.

Even if an application does check return values it might regard an error
return from EVP_DigestInit_ex() as a fatal error (it sometimes is: e.g. memory
allocation failure): it is unlikely to retry with a different algorithm.

The correct thing to do is never attempt to use the non-FIPS algorithm in
the first place. What that measn in practice is that while some applications
will "just work" when FIPS mode is enabled, others need to be modified to
correctly handle FIPS.

Steve.
--
Dr Stephen N. Henson. OpenSSL project core developer.
Commercial tech support now available see: http://www.openssl.org