[openssl-users] gpgsm/cms: int_rsa_verify:wrong signature length

[Viktor Dukhovni]

> On Apr 23, 2018, at 7:44 PM, Salz, Rich via openssl-users <openssl-users at openssl.org> wrote:
> 
> Your key starts with EF...  Since that has the high-bit on, it should be encoded with a leading zero.

Are you sure?  That does not look like a key to me...

More likely to be the signature block.  The broken one
is one byte shorter, and perhaps the issue is that the
RSA encrypted data has a leading byte that happens to
be zero (1 chance in ~<256 depending on high bytes of
modulus), and this got left out. Hence:

 int_rsa_verify:wrong signature length:../crypto/rsa/rsa_sign.c:132:

The code in question is:

   if (siglen != (size_t)RSA_size(rsa)) {
       RSAerr(RSA_F_INT_RSA_VERIFY, RSA_R_WRONG_SIGNATURE_LENGTH);
       return 0;
   }

which clearly shows that signature lengths are expected to be
exactly the same size as the modulus (padded with leading zeros
as needed).

Note that per:

   https://tools.ietf.org/html/rfc8017#section-8.2.1

the signature length MUST be the same as the modulus length.
A 256-byte modulus goes with a 2048-bit key.

-- 
	Viktor.