[openssl-dev] RSA SigVer (FIPS 186-4) Issue

[rsteck at symsysresearch.com]

I am getting incorrect False-Negative results when performing tests with 
186-4 vectors (generated by CAVS 17.6).

This vector is being reported false while CAVS says they should pass.

[mod = 1024]
n = 
d915e54ecbf96e1daadb5faa22856c4544a80c03d4cabeb58f7558a2ac9e939d387f86eebfa32aa81d6624def50684b46855a7cb86a15305ea84f34e9c18b1ca2b77e26f616f464cc675a9628aa2bc847c7a9f4ec2a3c49809901aa9ef76e5c779f621ddc791565708e65ea97a786653c745573c34310f135e29322d304fc009

SHAAlg = SHA512
e = 
00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001f2acb
Msg = 
912ca58670bda8a7d76c2f97bbeb76b1d795bc4ff2a6a29864a7d599d72dec984bc394a45d25ea71c8af28632c1484e90e53550c20e77848c33ef29acb6dd6da82226a4befda857158543b220545bce0474d7a66b92a8a81c62c4d216be84fb03e316fa2b044414e8f43b0fccfe5e83f896b738544d6a1ec74572e5740071fc2
S = 
122efd4d9f6dd746b959aebfe8e23fb0181afce8905cae19861da977aaff4c97792c488b065cc0a0f5c97f50f6545a77606d2e9b3e3533d3c54f6feb3670888238f3a58569a77f7a3178df599f637b13eaf13fa7a7706a7970aa3ab725b5e844d2861cc56cfed910328a8cf45b6f053e06f2b18c1a71ee48b38fed757a99b0c3

I've added some debug output and I get the following:

DIGEST = 
423d03646e5e4105181a5d9815b7fe3d588c3097ce7109ae4635add1be5ec026eb6860198914d1eb7fb1e62d86f60b3929fc6549d1b6b445ecc7b61219bf90d3
SIG = 
122EFD4D9F6DD746B959AEBFE8E23FB0181AFCE8905CAE19861DA977AAFF4C97792C488B065CC0A0F5C97F50F6545A77606D2E9B3E3533D3C54F6FEB3670888238F3A58569A77F7A3178DF599F637B13EAF13FA7A7706A7970AA3AB725B5E844D2861CC56CFED910328A8CF45B6F053E06F2B18C1A71EE48B38FED757A99B0C3
N   = 
D915E54ECBF96E1DAADB5FAA22856C4544A80C03D4CABEB58F7558A2AC9E939D387F86EEBFA32AA81D6624DEF50684B46855A7CB86A15305EA84F34E9C18B1CA2B77E26F616F464CC675A9628AA2BC847C7A9F4EC2A3C49809901AA9EF76E5C779F621DDC791565708E65EA97A786653C745573C34310F135E29322D304FC009
RR  = 
CE64B6D692597419FB9E6E4FF65B1A1181352AEF44565DDAC0F5F4C6B7E228853740B98FAAA513F4F3F4A42C908584496FF7E03D63E086AD23C6044F1506F98A23EB6BB31DD55E735C74EBDD17AB50DFB94008B2912B4CA77734DDC416866E5862E6C18DBF598BF243192FB1A657E5A4E5681DCBB34DF229D6136E0282AFEAC
SUB = 
CC2F99E162D3D6DC0B2178C5231FBAA42C94B954E08558D7E365F95641207114E50B7B55C4F8D968CE26DA9C2BFE2C6FD15629C7B0634A9B18489309AAC8423189392BB42F91F06590AE5AA4B928077680E69EC399910FCD921CCCCDAE0E7EE1F3C7B5C4EB9BBD97E4B4CBAE6012E7F978EED55F78FC2FF0C0C7FB4D0824C15D
IR  = 
CE64B6D692597419FB9E6E4FF65B1A1181352AEF44565DDAC0F5F4C6B7E228853740B98FAAA513F4F3F4A42C908584496FF7E03D63E086AD23C6044F1506F98A23EB6BB31DD55E735C74EBDD17AB50DFB94008B2912B4CA77734DDC416866E5862E6C18DBF598BF243192FB1A657E5A4E5681DCBB34DF229D6136E0282AFEAC
SigVer Result = 0

"RR" is the result of the power-E-mod-N operation, and "SUB" is the 
computed (N-RR) value.  "IR" is the choice of RR or (N-RR) used for the 
remainder of verification.

It seems clear to me that neither RR nor SUB has the correct form, and 
that this vector should not pass.

The lab is saying this must be a bug in OpenSSL, but no modification has 
been made that would introduce such a bug.  It's my contention that this 
is a bug in CAVS.  Can anyone shed some additional light onto the issue?

Thanks,
Randy Steck