Migrating to 3.x - how to handle PKEY types and Sig Algorithms?

[Dr. Pala]

Hello OpenSSL Community,

We are in the process of upgrading the LibPKI project to use the OSSL 
3.x instead of the OSSL 1.1.1x that we are using today ... and we can 
use some help. Specifically, we noticed some inconsistent behavior when 
porting from the 1.1.1x version and we hope you can */_help us finding 
what are we doing wrong and what is the right way of doing things_/* in 
the new "providers" world :)

The original code, when compiled against the OpenSSL 1.1.1x branch, 
works correctly... However, with the 3.x version, things do not behave 
the same way and our applications fail when compiled/linked against it. 
Our setting is OpenSSL 3.x with the OQS provider installed. The 
application successfully loads the 'default' provider first, then the 
'legacy', and finally the 'oqsprovider' one

Here's some of my initial questions:

  * *EVP_PKEY_id() and EVP_PKEY_type()* are not reliable anymore with
    providers (e.g., for "dilithium2" from the OQS provider they all
    return '0'). What shall we use to get the type of a PKEY structure
    (i.e., this is an "RSA" key or a "Falcon512" one) ? Currently, to
    address the issue with *EVP_PKEY_id*(), we are getting the name from
    the pkey (pkey_name = *EVP_PKEY_get0_type_name*(pkey) - Example
    Code:
    https://github.com/openca/libpki/blob/00b1fbb434dbf1ffd09fcad5ddfb7ceb7eeaba6a/src/openssl/pki_keypair.c#L350)
    and then converting that into an ID (*OBJ_sn2nid*(pkey_name)). If
    that is the supposed way of doing things (I really doubt it, we just
    used it as an hack), /_how do we get the type of key from the ID_
    (i.e., the functionality provided via*EVP_PKEY_type()*/) ? Shall
    we... string/mem compare when *EVP_PKEY_type*() returns 0? In other
    words, how do I know an EVP_KEY is an RSA one instead of a dilithium
    or falcon one (e.g., EVP_PKEY_type() vs. EVP_PKEY_EC, EVP_PKEY_RSA,
    etc.)? Example Code:
    https://github.com/openca/libpki/blob/28c510193735e6a68671bfb4cb4e66589a1fd9d0/src/drivers/hsm_main.c#L530

  * *OBJ_add_sigid() and OBJ_find_sigid_by_algs()* do not seem to work
    as usual. Specifically, with the above configuration (i.e., with the
    {default, legacy, oqsprovider} providers loaded), during the
    initialization of the library/application, we try to add a bunch of
    OIDs for new signature types that sometimes seem to persist and
    sometimes they do not. For example,  at startup _/we add a series of
    OIDs for hash-n-sign with dilithium2/_. We do this by creating the
    new signature OIDs with *OBJ_create*() first, and then we use those
    new IDs with the *OBJ_add_sigid*() with different hash identifiers.
    When we use the *OBJ_find_sigid_by_algs()* function right after
    adding the signature ID and, for some algorithms, the function fails
    to find it. For example, we can add a DILITHIUM2-SHA3_256 signature
    correctly, but the same code fails to retrieve the
    DILITHIUM2-SHA3_512. Even more strange is the fact that if we try to
    find sigid that were found during initialization, we now fail for no
    apparent reason. Example Code:
    https://github.com/openca/libpki/blob/28c510193735e6a68671bfb4cb4e66589a1fd9d0/src/openssl/pki_oid_defs.c#L622


  * *How to replace PKEY/ASN1_PKEY methods.* In our library, we
    currently provide a new type of algorithm (i.e., Composite Crypto)
    via a PKEY/ASN1_PKEY combination that we inject at initialization
    time via the *EVP_PKEY_meth_add0*() and *EVP_PKEY_asn1_add0*(). What
    is the equivalent functions to use under the providers' paradigm? In
    other words, do we need to generate a separate provider object or
    can we inject it programmatically? Example Code:
    https://github.com/openca/libpki/blob/28c510193735e6a68671bfb4cb4e66589a1fd9d0/src/openssl/composite/composite_init.c#L326

  * *EVP_PKEY_get0() fails when called from within the pkey_bits()
    function of the EVP_PKEY_ASN1_METHOD*. Specifically, when the
    pkey_bits() function is called with the parameter of type `const
    EVP_PKEY *pk`, the EVP_PKEY_get0(pk) function always returns NULL.
    What are we supposed to use to retrieve the internal key structure?
    Example Code:
    https://github.com/openca/libpki/blob/28c510193735e6a68671bfb4cb4e66589a1fd9d0/src/openssl/composite/composite_ameth.c#L963

The migration to the 3.x has some interesting differences that are quite 
challenging, thanks everybody for any help you can provide :)

Cheers,
Max

-- 
Best Regards,
Massimiliano Pala, Ph.D.
OpenCA Labs Director
OpenCA Logo
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mta.openssl.org/pipermail/openssl-users/attachments/20230821/b1b0ab42/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: iFz570LQ0K8ctlgY.png
Type: image/png
Size: 3146 bytes
Desc: not available
URL: <https://mta.openssl.org/pipermail/openssl-users/attachments/20230821/b1b0ab42/attachment-0001.png>