[openssl-users] Certificate verification fails with latest commits (ECDSA)

[Dr. Stephen Henson]

On Tue, Feb 03, 2015, jan.weil at ptb.de wrote:

> 
> This check fails for some of our certificates and the reason is that 
> openssl adds a padding byte for BIGNUMs in crypto/asn1/x_bignum.c if the 
> MSB is set. Our encoding does not contain these padding bytes and, 
> consequently, the re-encoded version of our certificate signature is two 
> bytes longer than before which results in an error.
> 
> RFC3279 defines
> 
>    Ecdsa-Sig-Value  ::=  SEQUENCE  {
>            r     INTEGER,
>            s     INTEGER  }
> 
> I've looked up the DER encoding rules for INTEGER here
> 
> http://www.itu.int/rec/T-REC-X.690-200811-I
> 
> and I can't find any evidence that this padding byte is mandatory. See 
> below for the relevant paragraph.
> 

The MSB is effectively a sign bit but the explanation in the standard isn't
very clear. If you take your example of GTS001.pem and do:

  openssl asn1parse -in GTS001.pem -strparse 367 -out sig.der

It will parse out the Ecdsa-Sig-Value field and you get:

    0:d=0  hl=2 l=  52 cons: SEQUENCE          
    2:d=1  hl=2 l=  24 prim: INTEGER           :-0739E1C1762E2E3E1D4480425633EA0BB669CE784DC3ACCB
   28:d=1  hl=2 l=  24 prim: INTEGER           :-332658917A3B05831D91176C0512CF91C617819E1A7CF14B

Note the two - signs.

Just to show it isn't just OpenSSL: if you use dumpasn1 on the output (sig.der)
you get:

  0  52: SEQUENCE {
  2  24:   INTEGER
       :     F8 C6 1E 3E 89 D1 D1 C1 E2 BB 7F BD A9 CC 15 F4
       :     49 96 31 87 B2 3C 53 35
       :     Error: Integer has a negative value.
 28  24:   INTEGER
       :     CC D9 A7 6E 85 C4 FA 7C E2 6E E8 93 FA ED 30 6E
       :     39 E8 7E 61 E5 83 0E B5
       :     Error: Integer has a negative value.
       :   }

>From your quote:

> The value of the 
> two's complement binary number is obtained by
> summing the numerical values assigned to each bit for those bits which are 
> set to one, excluding bit 8 of the first octet, and then
> reducing this value by the numerical value assigned to bit 8 of the first 
> octet if that bit is set to one.
> 

What this is saying is that if the MSB is one you subtract that value from
the result.

For example 0x80 without the MSB represents '0' the MSB represents 0x80 and
you subtract that resulting in -0x80. That's why you need the 0 padding byte
prepended if the MSB is one.

Steve.
--
Dr Stephen N. Henson. OpenSSL project core developer.
Commercial tech support now available see: http://www.openssl.org