[openssl/openssl] f00c5a: Add overflow checks to parse_number/parse_hex/pars...

[Neil Horman]

  Branch: refs/heads/openssl-3.0
  Home:   https://github.com/openssl/openssl
  Commit: f00c5a6076d79855112bddfcf7651ce3d4f8a6eb
      https://github.com/openssl/openssl/commit/f00c5a6076d79855112bddfcf7651ce3d4f8a6eb
  Author: Neil Horman <nhorman at openssl.org>
  Date:   2023-12-07 (Thu, 07 Dec 2023)

  Changed paths:
    M crypto/property/property_parse.c
    M test/property_test.c

  Log Message:
  -----------
  Add overflow checks to parse_number/parse_hex/parse_oct

Test the next arithmetic operation to safely determine if adding the
next digit in the passed property string will overflow

Also, noted a bug in the parse_hex code.  When parsing non-digit
characters (i.e. a-f and A-F), we do a tolower conversion (which is
fine), and then subtract 'a' to get the hex value from the ascii (which
is definately wrong).  We should subtract 'W' to convert tolower
converted hex digits in the range a-f to their hex value counterparts

Add tests to test_property_parse_error to ensure overflow checks work

Reviewed-by: Richard Levitte <levitte at openssl.org>
Reviewed-by: Tomas Mraz <tomas at openssl.org>
Reviewed-by: Tom Cosgrove <tom.cosgrove at arm.com>
(Merged from https://github.com/openssl/openssl/pull/22874)

(cherry picked from commit 986c48c4eb26861f25bc68ea252d8f2aad592735)