[openssl-users] FIPS Static Library linked into Win32 Dll builds but fails self test

[Jakob Bohm]

On 02/02/2016 20:03, Dr. Stephen Henson wrote:
> On Tue, Feb 02, 2016, Neptune wrote:
>
>> FIPS Object Module 2.0.9
>> OpenSSL 1.0.1l
>> Platform: Win32
>>
>> I am attempting to statically link a FIPS-capable library into a .dll. The
>> .dll is built without errors and by viewing the .dll in a hex editor I can
>> see the correct HMAC is embedded within and correct, but the self test is
>> failing.
>>
>> Originally I had built the FIPS-capable library as a dynamic library, but
>> during testing we experienced address clashes since the libeay32.dll
>> requires a fixed address and there is no way to guarantee an address we
>> choose will always be vacant, so static linking is a requirement.
>>
> You'll hit the same issue if you link the static version of OpenSSL into a
> DLL. The only way to escape it is to link the static version of OpenSSL into
> the executable or use Win64 which doesn't have this problem.
Actually, I think the issue (and solution) is simpler:

Due to the FIPS-HMAC test being affected by relocations, it will fail
in any DLL or EXE loaded at other than its default address.  Due to
the design of the amd64 (x86_64) instruction set, DLLs for that
architecture (which is not the only Win64 architecture) have much
fewer relocations, but detailed checking (which I have not done) is
needed to determine if the Windows compiler specified in the
validated FIPS policy will insert any such relocations within the
checksummed FIPS blob for Win64 on amd64.

Thus any DLL or EXE containing the FIPS module (and intending to use
it) must be linked with the "/FIXED" option (or have the relocations
stripped later in the build process), and its load address (BASE
address in Windows linker options) must be chosen so it is very
unlikely to clash with 3rd party DLLs, such as globally preloaded
antivirus DLLs.

For EXE files on Win32 (and Win64), the default base address for EXE
files is globally reserved and can be used.

For product-specific DLL files, that are "statically dynamically
linked" to the product EXE files (i.e. marked to be loaded at load
time by statically linking the EXE to a stub "import library"), base
addresses just after the end of the largest product EXE file are
generally safe too.  The easiest way to do this is to use the
sequence: REBASE (with proper file lists for and/or handling, also
specify extra space between modules corresponding to code signing
overhead) BIND, EDITBIN relocation stripping, fips-ld HMAC
calculation, actual release code signing, installer packing,
installer signing, release.

For DLL files that are not tied to a single product (such as the
default libcrypto DLLs, still named libeay on Windows) careful
selection of a likely free fixed load address is needed.

Enjoy

Jakob
-- 
Jakob Bohm, CIO, Partner, WiseMo A/S.  https://www.wisemo.com
Transformervej 29, 2860 Søborg, Denmark.  Direct +45 31 13 16 10
This public discussion message is non-binding and may contain errors.
WiseMo - Remote Service Management for PCs, Phones and Embedded