EVP_PKEY_ASN1_METHOD(3) conflicts

[Colton Jenkins]

I'm currently working on implementing EVP_PKEY_cmp for ruby's openssl
wrapper.

Initially, I was going by the documentation provided for EVP_PKEY_cmp, but
the maintainer discovered conflicting documentation.

https://www.openssl.org/docs/man1.1.1/man3/EVP_PKEY_ASN1_METHOD.html

States the following:
The param_cmp() method compares the parameters of keys *a* and *b*. It MUST
return 1 when the keys are equal, 0 when not equal, or a negative number on
error. It's called by EVP_PKEY_cmp_parameters(3)
<https://www.openssl.org/docs/man1.1.1/man3/EVP_PKEY_cmp_parameters.html>.

However
https://www.openssl.org/docs/man1.0.2/man3/EVP_PKEY_cmp.html

States something slightly different:
The function EVP_PKEY_cmp_parameters() and EVP_PKEY_cmp() return 1 if the
keys match, 0 if they don't match, -1 if the key types are different and -2
if the operation is not supported.

At this point I'm explicitly checking the type on each and not able to
check -2 against a 'not supported' or in ruby 'not implemented' exception
given EVP_PKEY_ASN1_METHOD(3) states ANY negative number can be an error.

The current work in progress can be found here if interested.
https://github.com/ruby/openssl/pull/383/files

Thanks,

Colton
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mta.openssl.org/pipermail/openssl-users/attachments/20200710/5a726909/attachment.html>