[openssl-users] Certificate verification fails with latest commits (ECDSA)

[Jakob Bohm]

Summary of thread so far: The latest security update enforces
that any inherently unsigned BIGNUM must be encoded as a non-
negative DER INTEGER (which has a leading 0 byte if the most
significant bit of the first byte would otherwise be set).

It is a well known historic bug that some other ASN.1
libraries incorrectly treat the DER/BER INTEGER type as
unsigned when encoding and decoding inherently unsigned
numbers, and that such libraries will thus accept the correct
encoding (leading 0 byte) as a non-canonical BER encoding
(and thankfully forget to normalize it to the wrong form),
while producing an incorrect encoding without the leading 0
byte.

Historically, OpenSSL (and probably some other ASN.1 libraries
too) have intentionally tolerated this specific incorrect
encoding, but the new security update now consistently rejects
it.  Would it reintroduce the related security issue to
explicitly tolerate the alternative encoding, perhaps by
allowing the EC code to accept negative numbers as their
unsigned encoding equivalents, while preserving the signed
form when round-tripping BER to BN to DER.  (This of cause
would still fail if the most significant 9 bits were all 1,
e.g. 0xFF8...., but that would still be 256 times rarer).

I am assuming without checking, that i2d_ASN1_INTEGER
already handles negative values.

Enjoy

Jakob
-- 
Jakob Bohm, CIO, Partner, WiseMo A/S.  http://www.wisemo.com
Transformervej 29, 2860 Søborg, Denmark.  Direct +45 31 13 16 10
This public discussion message is non-binding and may contain errors.
WiseMo - Remote Service Management for PCs, Phones and Embedded