Migrating to 3.x - how to handle PKEY types and Sig Algorithms?

Wed Aug 23 09:23:50 UTC 2023

On 22/08/2023 17:14, Dr. Pala wrote:
> Hi Matt, All,
> 
> thanks for the replies, I really appreciate the help :) Further 
> responses inline...
> 
> On 8/22/23 4:46 AM, Matt Caswell wrote:
>>
>> On 22/08/2023 06:20, Dr. Pala wrote:
>>> Hello OpenSSL Community,
>>>
>>> We are in the process of upgrading the LibPKI project to use the OSSL 
>>> 3.x instead of the OSSL 1.1.1x that we are using today ... and we can 
>>> use some help. Specifically, we noticed [...] falcon one (e.g., 
>>> EVP_PKEY_type() vs. EVP_PKEY_EC, EVP_PKEY_RSA, etc.)? Example Code: 
>>> https://github.com/openca/libpki/blob/28c510193735e6a68671bfb4cb4e66589a1fd9d0/src/drivers/hsm_main.c#L530
>>
>> We use EVP_PKEY_is_a() extensively internally to check whether a pkey 
>> is a particular type.
>>
>> https://www.openssl.org/docs/man3.1/man3/EVP_PKEY_is_a.html
> 
> Thanks! I guess that internally uses the same approach? How do you know 
> what strings to use for matching different algorithms? It would have 
> probably been useful to give these IDs some form of structure... maybe 
> they could be OIDs, instead of just "names" ? This would make things a 
> lot easier, I think, when you have multiple providers... I was really 
> bummed by the fact that we lost the ultimate NID/resource that was quite 
> unique and neat :) With this system that uses text we will have to start 
> "guessing" what is picked up... I fear it might lead to some 
> difficult-to-debug situations... :( I guess you went through all these 
> considerations in the past...
> 
> I also saw that there is a *EVP_PKEY_get0_type_name*() that provides the 
> "first key type name that is found" - is that preferred instead?

It is up to the provider to specify the name for the algorithm. For our 
own providers there is a naming structure that we try to use, but third 
party providers are not required to follow this:

https://github.com/openssl/openssl/blob/617cab094f0f0d4e71f8b9da5663be8ab06cba92/providers/defltprov.c#L74-L100

Each name can have multiple aliases associated with it and you can use 
any one of those aliases. Aliases tend to exist where we have used some 
other name in the past that doesn't necessarily conform to the 
"standard" name we try to use as per the link above, or where a name is 
in common usage. Many of our key types do also use the OID as an alias 
if one is available.

EVP_PKEY_get0_type_name() is the first name in the list of names we have 
for the key type so it is always possible to use that name to refer to it.

> 
>>>   * *OBJ_add_sigid() and OBJ_find_sigid_by_algs()* do not seem to work
>>>     as usual. Specifically, with the above configuration (i.e., with the
>>>    [...]
>>>     apparent reason. Example Code:
>>> https://github.com/openca/libpki/blob/28c510193735e6a68671bfb4cb4e66589a1fd9d0/src/openssl/pki_oid_defs.c#L622
>>
>> This sounds like it should work, so I don't know why you are seeing 
>> odd behaviour. Perhaps raise a github issue for this?
> 
> Ouch! I really hoped there was a new mechanism for registering IDs. The 
> interesting part is that the same code compiled against 1.1.1x works 
> correctly (i.e., the new sigid are found after being added), so I am 
> leaning toward a possible issue with the 3.x approach... I will try to 
> put together some code that reproduces the issue and submit it to the 
> repo... would it be ok if the example uses the oqsprovider? As I 
> understand, the issue should be there independently of the providers, 
> you can always register OIDs and add them to the stack of signature IDs.

Yes, its fine to use oqsprovider in any example - although obviously the 
simpler you can make it the better.

> 
>> One thing that does spring to mind is that I know that oqsprovider 
>> already registers various sigid OIDs - so plausibly there is some kind 
>> of issue caused by that:
>>
>> https://github.com/open-quantum-safe/oqs-provider/blob/b6da1e97b7180a779981d1b0bfa0ab2cd723066d/oqsprov/oqsprov.c#L592-L619
>>
>> https://github.com/open-quantum-safe/oqs-provider/blob/b6da1e97b7180a779981d1b0bfa0ab2cd723066d/oqsprov/oqsprov.c#L45-L71
> 
> Thanks for the links - I was wondering how they replaced the use of the 
> objects database to updating the OIDs internally. One thing that they 
> seem to do that I do not is the step with the assigning of the ID ... 
> and it seems that is needed to avoid spending too much time in lookup... 
> is it something that providers have to do or are encouraged to do? This 
> seems something that could potentially be done within the providers' 
> "glue" code?
> 

You mean the call to oqs_set_nid() in their code? I'm not familiar 
enough with oqsprovider to know how they are using that.

> 
> I will definitely start to look into that... thanks for the links. One 
> question I have is about "double dipping", so to speak. Would it be easy 
> to get the code from the BUILTIN version into a stand-alone provider? I 
> am thinking about this to see if it would be worth trying to provide 
> support for Composite Crypto in plain OpenSSL instead of having to link 
> the full LibPKI library if you need just the hybrid functionality ...

Yes this should be perfectly possible. We actually do it ourselves for 
the legacy provider (which is normally stand alone, but can become 
built-in if you configure OpenSSL with the "no-module" option.

There's a simpler example of this in the test framework where we have a 
test provider that can be either stand alone or built-in:

https://github.com/openssl/openssl/blob/openssl-3.1/test/p_test.c

> 
>>>   * *EVP_PKEY_get0() fails when called from within the pkey_bits()
>>>     function of the EVP_PKEY_ASN1_METHOD*. Specifically, when the
>>>     pkey_bits() function is called with the parameter of type `const
>>>     EVP_PKEY *pk`, the EVP_PKEY_get0(pk) function always returns NULL.
>>>     What are we supposed to use to retrieve the internal key structure?
>>>     Example Code:
>>> https://github.com/openca/libpki/blob/28c510193735e6a68671bfb4cb4e66589a1fd9d0/src/openssl/composite/composite_ameth.c#L963
>>>
>>
>> EVP_PKEY_get0() returns a pointer to the legacy key structure (e.g. 
>> RSA, DSA, DH, EC_KEY, etc). It won't work if you call it on a provider 
>> based EVP_PKEY (returns NULL) but you can call the typed equivalents, 
>> e.g. EVP_PKEY_get0_RSA(). In that case it gives you a cached *copy* of 
>> the legacy key (the real key is held in the provider and is 
>> inaccessibly directly to the application).
> 
> Actually, this is related to the PKEY_METH/ASN1_PKEY_METH that we 
> develop to support Composite Crypto. In there, we need to get the 
> reference to the internal data structure of the key to retrieve the 
> stack of keys - for example, when we want to call the *EVP_PKEY_bits*(), 
> which, in turn, calls the ASN1_PKEY_METH's *pkey_bits*() function. In 
> there, we use the *EVP_PKEY_get0*() function to retrieve the internal 
> key structure (COMPOSITE_KEY *). The problem is that EVP_PKEY_get0() 
> returns NULL and, therefore, fails. The key is loaded by the method and 
> the internal structure is assigned with the EVP_PKEY_assign_COMPOSITE() 
> function which, in turn, calls the *EVP_PKEY_assign*() - see the code here:
> 
>   * https://github.com/openca/libpki/blob/33464662fd26f7ee473ca3f5d0d3d1c71f6dd0de/src/openssl/composite/composite_utils.c#L45
> 
>> It's unclear to me from your question whether you are talking about 
>> your *own* EVP_PKEY type, i.e. one you've defined yourself that does 
>> not use a provider. If so then this should still work as it did 
>> before, i.e. as long as data has been assigned to the key via 
>> EVP_PKEY_assign() then it should be available via a subsequent call to 
>> EVP_PKEY_get0().
> 
> Well... I guess it should work, but it does not :(
> 
> I am not sure what is going on, but there are definitely differences 
> with how the code works in 1.1.1x - I hoped for some additional 
> procedures we might needed to take... but it seems we are heading 
> towards multiple "beautiful" debugging session(s)... maybe the issue is 
> still related to OIDs/IDs... ? We'll see... if I understand what's going 
> on I will submit an Issue in GitHub... but I hope the issue is in our 
> code :)

If you think it is an OpenSSL problem, just raise a github issue and we 
can take a look.

Matt

> 
> Thanks again Matt (and the community), I hope these considerations and 
> questions might help other people who are in the middle of migrating. 
> Thanks!
> 
> Cheers,
> Max
> 
> -- 
> Best Regards,
> Massimiliano Pala, Ph.D.
> OpenCA Labs Director
> OpenCA Logo