[openssl/openssl] b1b3cb: cms: avoid intermittent test failure

[James Muir]

  Branch: refs/heads/openssl-3.1
  Home:   https://github.com/openssl/openssl
  Commit: b1b3cb9d9a14b630130181ee0c8801752a74e71c
      https://github.com/openssl/openssl/commit/b1b3cb9d9a14b630130181ee0c8801752a74e71c
  Author: James Muir <james at openssl.org>
  Date:   2023-12-18 (Mon, 18 Dec 2023)

  Changed paths:
    M test/recipes/80-test_cms.t
    A test/smime-certs/smrsa3-cert.pem
    A test/smime-certs/smrsa3-key.pem

  Log Message:
  -----------
  cms: avoid intermittent test failure

If you decrypt a random input using RSAES-PKCS-v1_5, then there is a
non-negligible chance that the result will look like a valid plaintext
(that is why RSAES-PKCS-v1_5 shouldn't be used anymore).  This was the
cause of an intermittent failure in a test that did a cms-encrypt
operation targetting multiple recipients.

The failure happened during key-only decrypt.  The recipient decrypts
every RSA ciphertext -- only one is supposed to decrypt successfully,
which would reveal the right content-key.  Occassionally, more than
one decrypted successfully.

Update the test by specifying the recipient cert in the decrypt op
(this avoids looping over all RSA ciphertexts).

Add a new test to get coverage for key-only decrypt, but use RSA-OAEP
during the encrypt op.

Fixes https://github.com/openssl/project/issues/380

Testing:

  $ make TESTS='test_cms' test

Reviewed-by: Neil Horman <nhorman at openssl.org>
Reviewed-by: Tomas Mraz <tomas at openssl.org>
(Merged from https://github.com/openssl/openssl/pull/23055)

(cherry picked from commit ffed597882baf2f07274e7eaa8f3c4fa9fa74ac1)