openssl 3.0 - id2_x509() now fails

[Ken Goldman]

On 8/9/2021 3:50 AM, Tomas Mraz wrote:
> On Fri, 2021-08-06 at 18:06 -0400, Ken Goldman wrote:
>> On 8/6/2021 1:11 PM, Ken Goldman wrote:
>>> I have an application where I have to create a partial x509
>>> certificate.  It gets sent to an HSM, which fills in the public key
>>> and signs it.
>>>
>>> I was calling
>>>
>>>       X509_new
>>>       X509_set_version
>>>       X509_set_issuer_name
>>>       X509_get_notBefore
>>>       X509_get_notAfter
>>>       X509_set_subject_name
>>>       X509_EXTENSION_create_by_OBJ
>>>
>>> and then
>>>       i2d_x509
>>> to send the serialized partial certificate to the HSM.
>>>
>>> This worked in 1.0.1, 1.0.2, 1.1.1, but fails in 3.0.0.
>>>
>>> In debugging, even this fails.
>>>
>>>       X509_new
>>>       i2d_x509
>>>
>>> Suggestions?
>>
>> Following up, I found that just omitting the signature from the
>> X509 structure causes i2d_x509 to fail.
>>
>> I tried i2d_re_X509_tbs(), but it also failed.
> 
> I am afraid with the current 3.0 codebase there are not many options
> how to workaround apart from either signing the certificate with a
> bogus key - if the HSM is able to re-sign such certificate.

My hope is that the maintainers will revert this change.  Perhaps
they can write a new variant of i2d_x509 that requires the full
certificate rather than change the existing API.

The i2d__re_x509_tbs() API seems promising (tbs is 'to be signed'),
but it apparently is strict on what data must be there.

The HSM (TPM, ISO 11889) cannot change.  It expects a
partial certificate.  It's API is already defined.

> Another (more complicated) option would be to define your own ASN.1
> X509 structure where the signature would be optional and thus the
> stricter encoder that is now in 3.0 codebase would allow encoding the
> incomplete certificate.

If you can post some hints on how to do this, I'll try it.

My alternative is to write the asn1 code from scratch, but I know
how fragile that will be.