Migrating to 3.x - how to handle PKEY types and Sig Algorithms?

Tue Aug 22 16:14:00 UTC 2023

Hi Matt, All,

thanks for the replies, I really appreciate the help :) Further 
responses inline...

On 8/22/23 4:46 AM, Matt Caswell wrote:
>
> On 22/08/2023 06:20, Dr. Pala wrote:
>> Hello OpenSSL Community,
>>
>> We are in the process of upgrading the LibPKI project to use the OSSL 
>> 3.x instead of the OSSL 1.1.1x that we are using today ... and we can 
>> use some help. Specifically, we noticed [...] falcon one (e.g., 
>> EVP_PKEY_type() vs. EVP_PKEY_EC, EVP_PKEY_RSA, etc.)? Example Code: 
>> https://github.com/openca/libpki/blob/28c510193735e6a68671bfb4cb4e66589a1fd9d0/src/drivers/hsm_main.c#L530
>
> We use EVP_PKEY_is_a() extensively internally to check whether a pkey 
> is a particular type.
>
> https://www.openssl.org/docs/man3.1/man3/EVP_PKEY_is_a.html

Thanks! I guess that internally uses the same approach? How do you know 
what strings to use for matching different algorithms? It would have 
probably been useful to give these IDs some form of structure... maybe 
they could be OIDs, instead of just "names" ? This would make things a 
lot easier, I think, when you have multiple providers... I was really 
bummed by the fact that we lost the ultimate NID/resource that was quite 
unique and neat :) With this system that uses text we will have to start 
"guessing" what is picked up... I fear it might lead to some 
difficult-to-debug situations... :( I guess you went through all these 
considerations in the past...

I also saw that there is a *EVP_PKEY_get0_type_name*() that provides the 
"first key type name that is found" - is that preferred instead?

>>   * *OBJ_add_sigid() and OBJ_find_sigid_by_algs()* do not seem to work
>>     as usual. Specifically, with the above configuration (i.e., with the
>>    [...]
>>     apparent reason. Example Code:
>> https://github.com/openca/libpki/blob/28c510193735e6a68671bfb4cb4e66589a1fd9d0/src/openssl/pki_oid_defs.c#L622
>
> This sounds like it should work, so I don't know why you are seeing 
> odd behaviour. Perhaps raise a github issue for this?

Ouch! I really hoped there was a new mechanism for registering IDs. The 
interesting part is that the same code compiled against 1.1.1x works 
correctly (i.e., the new sigid are found after being added), so I am 
leaning toward a possible issue with the 3.x approach... I will try to 
put together some code that reproduces the issue and submit it to the 
repo... would it be ok if the example uses the oqsprovider? As I 
understand, the issue should be there independently of the providers, 
you can always register OIDs and add them to the stack of signature IDs.

> One thing that does spring to mind is that I know that oqsprovider 
> already registers various sigid OIDs - so plausibly there is some kind 
> of issue caused by that:
>
> https://github.com/open-quantum-safe/oqs-provider/blob/b6da1e97b7180a779981d1b0bfa0ab2cd723066d/oqsprov/oqsprov.c#L592-L619 
>
>
> https://github.com/open-quantum-safe/oqs-provider/blob/b6da1e97b7180a779981d1b0bfa0ab2cd723066d/oqsprov/oqsprov.c#L45-L71 
>

Thanks for the links - I was wondering how they replaced the use of the 
objects database to updating the OIDs internally. One thing that they 
seem to do that I do not is the step with the assigning of the ID ... 
and it seems that is needed to avoid spending too much time in lookup... 
is it something that providers have to do or are encouraged to do? This 
seems something that could potentially be done within the providers' 
"glue" code?

>>
>>   * *How to replace PKEY/ASN1_PKEY methods.* In our library, we
>>     currently provide a new type of algorithm (i.e., Composite Crypto)
>> [...]    can we inject it programmatically? Example Code:
>> https://github.com/openca/libpki/blob/28c510193735e6a68671bfb4cb4e66589a1fd9d0/src/openssl/composite/composite_init.c#L326
>
> Yes, you would need a new provider to add a new algorithm (although 
> the old approach should still *work* - even though it is deprecated). 
> You can just add a "built-in" one if you want (so no need to create a 
> separate .so file for it - it can just live in your application code)
>
> https://www.openssl.org/docs/man3.1/man3/OSSL_PROVIDER_add_builtin.html

Thanks, that is exactly what I was looking for...

>
> Start by looking at the page for general information for provider 
> authors:
>
> https://www.openssl.org/docs/man3.1/man7/provider.html
>
> Richard Levitte has a "toy" provider example which you can look at here:
>
> https://github.com/provider-corner/vigenere/blob/main/vigenere.c
>
Great! That is really useful - learning by example :)

> To implement an algorithm yourself you need to first determine what 
> kind of algorithm it is (e.g. cipher, signature, key-exchange, etc) 
> and then implement the appropriate provider functions for that 
> algorithm. To implement a new signature algorithm you would need to 
> implement the signature functions as well as the key management 
> functions. E.g. see
>
> https://www.openssl.org/docs/man3.1/man7/provider-signature.html
>
> and
>
> https://www.openssl.org/docs/man3.1/man7/provider-keymgmt.html

I will definitely start to look into that... thanks for the links. One 
question I have is about "double dipping", so to speak. Would it be easy 
to get the code from the BUILTIN version into a stand-alone provider? I 
am thinking about this to see if it would be worth trying to provide 
support for Composite Crypto in plain OpenSSL instead of having to link 
the full LibPKI library if you need just the hybrid functionality ...

>>   * *EVP_PKEY_get0() fails when called from within the pkey_bits()
>>     function of the EVP_PKEY_ASN1_METHOD*. Specifically, when the
>>     pkey_bits() function is called with the parameter of type `const
>>     EVP_PKEY *pk`, the EVP_PKEY_get0(pk) function always returns NULL.
>>     What are we supposed to use to retrieve the internal key structure?
>>     Example Code:
>> https://github.com/openca/libpki/blob/28c510193735e6a68671bfb4cb4e66589a1fd9d0/src/openssl/composite/composite_ameth.c#L963
>>
>
> EVP_PKEY_get0() returns a pointer to the legacy key structure (e.g. 
> RSA, DSA, DH, EC_KEY, etc). It won't work if you call it on a provider 
> based EVP_PKEY (returns NULL) but you can call the typed equivalents, 
> e.g. EVP_PKEY_get0_RSA(). In that case it gives you a cached *copy* of 
> the legacy key (the real key is held in the provider and is 
> inaccessibly directly to the application).

Actually, this is related to the PKEY_METH/ASN1_PKEY_METH that we 
develop to support Composite Crypto. In there, we need to get the 
reference to the internal data structure of the key to retrieve the 
stack of keys - for example, when we want to call the *EVP_PKEY_bits*(), 
which, in turn, calls the ASN1_PKEY_METH's *pkey_bits*() function. In 
there, we use the *EVP_PKEY_get0*() function to retrieve the internal 
key structure (COMPOSITE_KEY *). The problem is that EVP_PKEY_get0() 
returns NULL and, therefore, fails. The key is loaded by the method and 
the internal structure is assigned with the EVP_PKEY_assign_COMPOSITE() 
function which, in turn, calls the *EVP_PKEY_assign*() - see the code here:

  * https://github.com/openca/libpki/blob/33464662fd26f7ee473ca3f5d0d3d1c71f6dd0de/src/openssl/composite/composite_utils.c#L45

> It's unclear to me from your question whether you are talking about 
> your *own* EVP_PKEY type, i.e. one you've defined yourself that does 
> not use a provider. If so then this should still work as it did 
> before, i.e. as long as data has been assigned to the key via 
> EVP_PKEY_assign() then it should be available via a subsequent call to 
> EVP_PKEY_get0().

Well... I guess it should work, but it does not :(

I am not sure what is going on, but there are definitely differences 
with how the code works in 1.1.1x - I hoped for some additional 
procedures we might needed to take... but it seems we are heading 
towards multiple "beautiful" debugging session(s)... maybe the issue is 
still related to OIDs/IDs... ? We'll see... if I understand what's going 
on I will submit an Issue in GitHub... but I hope the issue is in our 
code :)

Thanks again Matt (and the community), I hope these considerations and 
questions might help other people who are in the middle of migrating. 
Thanks!

Cheers,
Max

-- 
Best Regards,
Massimiliano Pala, Ph.D.
OpenCA Labs Director
OpenCA Logo
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mta.openssl.org/pipermail/openssl-users/attachments/20230822/0e2b2ecc/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: sJJ9sLHvLLuZQBik.png
Type: image/png
Size: 3146 bytes
Desc: not available
URL: <https://mta.openssl.org/pipermail/openssl-users/attachments/20230822/0e2b2ecc/attachment-0001.png>