[openssl/openssl] 5a53d7: Fix infinite loops in DSA sign code.

[Shane]

  Branch: refs/heads/openssl-3.0
  Home:   https://github.com/openssl/openssl
  Commit: 5a53d73f1f97684a60ab4a80058c7da16b744f54
      https://github.com/openssl/openssl/commit/5a53d73f1f97684a60ab4a80058c7da16b744f54
  Author: slontis <shane.lontis at oracle.com>
  Date:   2023-03-01 (Wed, 01 Mar 2023)

  Changed paths:
    M crypto/dsa/dsa_err.c
    M crypto/dsa/dsa_ossl.c
    M crypto/err/openssl.txt
    M include/crypto/dsaerr.h
    M include/openssl/dsaerr.h
    M test/dsatest.c

  Log Message:
  -----------
  Fix infinite loops in DSA sign code.

Fixes #20268

Values such as q=1 or priv=0 caused infinite loops when calling
DSA_sign() without these changes.

There are other cases where bad domain parameters may have caused
infinite loops where the retry counter has been added. The simpler case
of priv=0 also hits this case. q=1 caused an infinite loop in the setup.

The max retry value has been set to an arbitrary value of 8 (it is
unlikely to ever do a single retry for valid values).

The minimum q bits was set to an arbitrary value of 128 (160 is still
used for legacy reasons when using 512 bit keys).

Thanks @guidovranken for detecting this, and @davidben for his
insightful analysis.

Reviewed-by: Matt Caswell <matt at openssl.org>
Reviewed-by: Paul Dale <pauli at openssl.org>
(Merged from https://github.com/openssl/openssl/pull/20384)

(cherry picked from commit 3a4e09ab42654b3d223f0f8dd1a9c58b2902ddcc)