[openssl-users] FIPS Static Library linked into Win32 Dll builds but fails self test

[Neptune]

FIPS Object Module 2.0.9 
OpenSSL 1.0.1l 
Platform: Win32 

I am attempting to statically link a FIPS-capable library into a .dll. The
.dll is built without errors and by viewing the .dll in a hex editor I can
see the correct HMAC is embedded within and correct, but the self test is
failing. 

Originally I had built the FIPS-capable library as a dynamic library, but
during testing we experienced address clashes since the libeay32.dll
requires a fixed address and there is no way to guarantee an address we
choose will always be vacant, so static linking is a requirement. 

Here is my process... 
1. Build the .dll project in Visual Studio 2005 
2. Run a custom batch file which links all of the .obj files including the
fips_premain.obj 

Here is my batch file: 
<<<< 
@ECHO OFF 
SET FIPS_PATH=C:\SWTOOLS\OpenSSL_FIPS\openssl-fips-2.0.9 
SET INC_D=C:\SWTOOLS\OpenSSL_FIPS\openssl-1.0.1l\inc32 
SET INCL_D=C:\SWTOOLS\OpenSSL_FIPS\openssl-1.0.1l\tmp32 

SET INC=-I %INC_D% -I %INCL_D% 
SET FIPS_CC=cl 
SET CFLAG=/MD /Ox -DOPENSSL_THREADS -DDSO_WIN32 -W3 -Gs0 -Gy -nologo
-DOPENSSL_SYSNAME_WIN32 -DWIN32_LEAN_AND_MEAN -DL_ENDIAN -DUNICODE
-D_UNICODE -D_CRT_SECURE_NO_DEPRECATE -DOPENSSL_IA32_SSE2
-DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m
-IC:\SWTOOLS\OpenSSL_FIPS\openssl-fips-2.0.9\include -DSHA1_ASM -DSHA256_ASM
-DSHA512_ASM -DMD5_ASM -DAES_ASM -DVPAES_ASM -DBSAES_ASM -DWHIRLPOOL_ASM
-DGHASH_ASM -DOPENSSL_NO_RC5 -DOPENSSL_NO_MD2 -DOPENSSL_NO_KRB5
-DOPENSSL_FIPS -DOPENSSL_NO_JPAKE -DOPENSSL_NO_DYNAMIC_ENGINE 
SET LIB_CFLAG= /Zl /Zi 
SET SHLIB_CFLAG= 
SET SHLIB_CFLAGS=%INC% %CFLAG% %LIB_CFLAG% %SHLIB_CFLAG% 
SET
FIPS_CC_ARGS=/FoC:\SWTOOLS\OpenSSL_FIPS\Static_Libraries\luaCrypto\fips_premain.obj
%SHLIB_CFLAGS% -c 

SET FIPS_LINK=link 
SET
PREMAIN_DSO_EXE=C:\SWTOOLS\OpenSSL_FIPS\openssl-1.0.1l\out32\fips_premain_dso.exe 
SET
FIPS_TARGET=C:\SWTOOLS\OpenSSL_FIPS\Static_Libraries\luaCrypto\luaCrypto.dll 
SET FIPS_SHA1_EXE=%FIPS_PATH%\bin\fips_standalone_sha1.exe 
SET FIPS_SIG=perl C:\SWTOOLS\OpenSSL_FIPS\openssl-fips-2.0.9\util\msincore 
SET FIPSLIB_D=%FIPS_PATH%\lib 

@ECHO ON 
perl %FIPS_PATH%\bin\fipslink.pl /MACHINE:X86 /ERRORREPORT:PROMPT /DEBUG
/DLL /FIXED /NOLOGO /MAPINFO:EXPORTS /SUBSYSTEM:WINDOWS
/OUT:"C:\SWTOOLS\OpenSSL_FIPS\Static_Libraries\luaCrypto\luaCrypto.dll"
/LIBPATH:\"C:\SWTOOLS\OpenSSL_FIPS\Static_Libraries\luaCrypto\lib\"
/LIBPATH:\"C:\SWTOOLS\OpenSSL_FIPS\openssl-fips-2.0.9\lib\"
/LIBPATH:\"C:\SWTOOLS\OpenSSL_FIPS\openssl-1.0.1l\out32\" libeayfips32.lib
ssleay32.lib libeaycompat32.lib ws2_32.lib gdi32.lib advapi32.lib
crypt32.lib user32.lib kernel32.lib fiblua.lib rijndael.lib winspool.lib
comdlg32.lib shell32.lib ole32.lib oleaut32.lib uuid.lib odbc32.lib
odbccp32.lib CryptoHelper.obj luaCrypto.obj SHA1Ex.obj fips_premain.obj 
@ECHO OFF 
>>>> 

So, this process generates a .dll which can be loaded, but when
FIPS_set_mode(1); is called, the self-test fails with bad fingerprint. 

I know the HMAC is in there, as verified in hex editor, so I'm thinking this
must have something to do with the location of the HMAC, but how can I have
any control over where it is place? 

Thanks for any help! 

Paul 



--
View this message in context: http://openssl.6102.n7.nabble.com/FIPS-Static-Library-linked-into-Win32-Dll-builds-but-fails-self-test-tp63011.html
Sent from the OpenSSL - User mailing list archive at Nabble.com.