[openssl/openssl] fea151: Obtain PSS salt length from provider

[Clemens Lang]

  Branch: refs/heads/openssl-3.1
  Home:   https://github.com/openssl/openssl
  Commit: fea151a24de59f546e6d485f352c309953f38e1d
      https://github.com/openssl/openssl/commit/fea151a24de59f546e6d485f352c309953f38e1d
  Author: Clemens Lang <cllang at redhat.com>
  Date:   2022-12-09 (Fri, 09 Dec 2022)

  Changed paths:
    M crypto/cms/cms_rsa.c
    M crypto/rsa/rsa_ameth.c
    M test/recipes/15-test_rsapss.t
    M test/recipes/80-test_cms.t

  Log Message:
  -----------
  Obtain PSS salt length from provider

Rather than computing the PSS salt length again in core using
ossl_rsa_ctx_to_pss_string, which calls rsa_ctx_to_pss and computes the
salt length, obtain it from the provider using the
OSSL_SIGNATURE_PARAM_ALGORITHM_ID param to handle the case where the
interpretation of the magic constants in the provider differs from that
of OpenSSL core.

Add tests that verify that the rsa_pss_saltlen:max,
rsa_pss_saltlen:<integer> and rsa_pss_saltlen:digest options work and
put the computed digest length into the CMS_ContentInfo struct when
using CMS. Do not add a test for the salt length generated by a provider
when no specific rsa_pss_saltlen option is defined, since that number
could change between providers and provider versions, and we want to
preserve compatibility with older providers.

Signed-off-by: Clemens Lang <cllang at redhat.com>

Reviewed-by: Dmitry Belyavskiy <beldmit at gmail.com>
Reviewed-by: Tomas Mraz <tomas at openssl.org>

(cherry picked from commit 5a3bbe1712435d577bbc5ec046906979e8471d8b)

(Merged from https://github.com/openssl/openssl/pull/19862)


  Commit: db2fc00945bd128b873d9f66944dfde49f8a3853
      https://github.com/openssl/openssl/commit/db2fc00945bd128b873d9f66944dfde49f8a3853
  Author: Clemens Lang <cllang at redhat.com>
  Date:   2022-12-09 (Fri, 09 Dec 2022)

  Changed paths:
    M CHANGES.md
    M crypto/rsa/rsa_ameth.c
    M crypto/rsa/rsa_pss.c
    M doc/man3/EVP_PKEY_CTX_ctrl.pod
    M doc/man7/EVP_SIGNATURE-RSA.pod
    M include/openssl/core_names.h
    M include/openssl/rsa.h
    M providers/implementations/signature/rsa_sig.c
    M test/recipes/15-test_rsapss.t
    M test/recipes/25-test_req.t
    M test/recipes/80-test_cms.t
    M test/smime-certs/mksmime-certs.sh
    A test/smime-certs/smrsa1024.pem

  Log Message:
  -----------
  signature: Clamp PSS salt len to MD len

FIPS 186-4 section 5 "The RSA Digital Signature Algorithm", subsection
5.5 "PKCS #1" says: "For RSASSA-PSS […] the length (in bytes) of the
salt (sLen) shall satisfy 0 <= sLen <= hLen, where hLen is the length of
the hash function output block (in bytes)."

Introduce a new option RSA_PSS_SALTLEN_AUTO_DIGEST_MAX and make it the
default. The new value will behave like RSA_PSS_SALTLEN_AUTO, but will
not use more than the digest length when signing, so that FIPS 186-4 is
not violated. This value has two advantages when compared with
RSA_PSS_SALTLEN_DIGEST: (1) It will continue to do auto-detection when
verifying signatures for maximum compatibility, where
RSA_PSS_SALTLEN_DIGEST would fail for other digest sizes. (2) It will
work for combinations where the maximum salt length is smaller than the
digest size, which typically happens with large digest sizes (e.g.,
SHA-512) and small RSA keys.

J.-S. Coron shows in "Optimal Security Proofs for PSS and Other
Signature Schemes. Advances in Cryptology – Eurocrypt 2002, volume 2332
of Lecture Notes in Computer Science, pp. 272 – 287. Springer Verlag,
2002." that longer salts than the output size of modern hash functions
do not increase security: "For example,for an application in which at
most one billion signatures will be generated, k0 = 30 bits of random
salt are actually sufficient to guarantee the same level of security as
RSA, and taking a larger salt does not increase the security level."

Signed-off-by: Clemens Lang <cllang at redhat.com>

Reviewed-by: Dmitry Belyavskiy <beldmit at gmail.com>
Reviewed-by: Tomas Mraz <tomas at openssl.org>

(cherry picked from commit 6c73ca4a2f4ea71f4a880670624e7b2fdb6f32da)

(Merged from https://github.com/openssl/openssl/pull/19862)


Compare: https://github.com/openssl/openssl/compare/3956fc91afc9...db2fc00945bd