[openssl/openssl] 3a4e09: Fix infinite loops in DSA sign code.

[Shane]

  Branch: refs/heads/master
  Home:   https://github.com/openssl/openssl
  Commit: 3a4e09ab42654b3d223f0f8dd1a9c58b2902ddcc
      https://github.com/openssl/openssl/commit/3a4e09ab42654b3d223f0f8dd1a9c58b2902ddcc
  Author: slontis <shane.lontis at oracle.com>
  Date:   2023-03-01 (Wed, 01 Mar 2023)

  Changed paths:
    M crypto/dsa/dsa_err.c
    M crypto/dsa/dsa_ossl.c
    M crypto/err/openssl.txt
    M include/crypto/dsaerr.h
    M include/openssl/dsaerr.h
    M test/dsatest.c

  Log Message:
  -----------
  Fix infinite loops in DSA sign code.

Fixes #20268

Values such as q=1 or priv=0 caused infinite loops when calling
DSA_sign() without these changes.

There are other cases where bad domain parameters may have caused
infinite loops where the retry counter has been added. The simpler case
of priv=0 also hits this case. q=1 caused an infinite loop in the setup.

The max retry value has been set to an arbitrary value of 8 (it is
unlikely to ever do a single retry for valid values).

The minimum q bits was set to an arbitrary value of 128 (160 is still
used for legacy reasons when using 512 bit keys).

Thanks @guidovranken for detecting this, and @davidben for his
insightful analysis.

Reviewed-by: Matt Caswell <matt at openssl.org>
Reviewed-by: Paul Dale <pauli at openssl.org>
(Merged from https://github.com/openssl/openssl/pull/20384)


  Commit: 5f820bd7535b871fdfdc0303c3af23ba4be901f0
      https://github.com/openssl/openssl/commit/5f820bd7535b871fdfdc0303c3af23ba4be901f0
  Author: slontis <shane.lontis at oracle.com>
  Date:   2023-03-01 (Wed, 01 Mar 2023)

  Changed paths:
    M crypto/ec/ec_err.c
    M crypto/ec/ecdsa_ossl.c
    M crypto/err/openssl.txt
    M include/crypto/ecerr.h
    M include/openssl/ecerr.h

  Log Message:
  -----------
  Fix potential infinite loops in ECDSA signing.

Similiar checks to the DSA code have been added for ECDSA also.
This should not be a problem when using named groups.

Reviewed-by: Matt Caswell <matt at openssl.org>
Reviewed-by: Paul Dale <pauli at openssl.org>
(Merged from https://github.com/openssl/openssl/pull/20384)


Compare: https://github.com/openssl/openssl/compare/a6d7093a1dc8...5f820bd7535b