Support for retrieving hash algorithm from RSA signature before/during verification

[Atul Singh]

RSASSA-PKCS1-v1_5 signature scheme encodes the AlgorithmIdentifier of
Hash Function in the message data before passing it through the
encryption[1]. So, technically one doesn't need to know the Hash
Function beforehand for verification purposes -- it can be decoded
right out of the decrypted octets.

Does openssl provide a way to perform verification without the
application supplying the Hash Function for RSASSA-PKCS1-v1_5
signatures? I have played around with EVP_DigestVerifyInit and
friends, but it doesn't seem to be possible -- The Hash Function is
required at the init context stage itself and there doesn't seem to be
any way around it[2].

I also did not see anything obvious in rsa.h that could just return
the AlgorithmIdentifier that can be used for EVP_DigestVerifyInit
later.

The reason I am asking this question is that IKEv2 leaves it to the
implementation to choose their Authentication method unannounced[3].
If an IKE peer chooses RSA Signature as their authentication method,
it can then go on to use any of the Hash Functions supported by RSA
(RFC5996 only recommends making SHA1 default). This is causing a lot
of incompatibility issues between vendors. Technically, one could try
all possible algorithms one by one but that means doing 4 passes for
SHA1, SHA256, SHA384 and SHA512 (assuming MD2 and MD5 are not in use
anymore).

Regards,
Atul

[1] https://datatracker.ietf.org/doc/html/rfc3447#section-9.2
[2] https://www.openssl.org/docs/man1.1.1/man3/EVP_DigestInit.html
[2] https://datatracker.ietf.org/doc/html/rfc5996#section-3.8