issue with X509_issuer_and_serial_hash returning different values under OpenSSL 3

[adv2011 at rustichelli.net]

UPDATE: I now compile a version of the code that replaces all of the 
pointers, but still I don't get the result as from OpenSSL 1.
This is the current exceprt of interest... from here, I'm stuck:

     // cannot do this under OpenSSL 3: f = 
X509_NAME_oneline(a->cert_info.issuer, NULL, 0);
         f = X509_NAME_oneline(X509_get_issuer_name(a), NULL, 0);
     [...]
         if (!EVP_DigestUpdate
         // cannot do this under OpenSSL 3: (ctx, (unsigned char 
*)a->cert_info.serialNumber.data,
                 (ctx, ASN1_STRING_data(X509_get_serialNumber(a)),
         // OpenSSL 1: (unsigned long)a->cert_info.serialNumber.length))
                 (unsigned 
long)ASN1_STRING_length(X509_get_serialNumber(a))))

What am I doing wrong?

On 3/8/23 10:55, adv2011 at rustichelli.net wrote:
> (reposted with the right subject, sorry)
>
> Hi all, I am starting to port some code to OpenSSL 3 (it's my first 
> taste of it), and I'm stuck with a problem. I'm working under Ubuntu 22.
>
> I saw that the function X509_issuer_and_serial_hash doesn't return the 
> same value it did before (though not for an obvious reason), and since 
> that value is used by my software to identify some certificates 
> against a DB, I need to replicate the old behaviour.
>
> To do so, I'm first trying to change the old function (from OpenSSL 
> 1.1) so that it compiles under OpenSSL 3.
>
> Here, a is of type X509, I always accessed most data from pointers. 
> Now that they are gone, how do I read the following information to 
> obtain exactly the same data?
>
> - a->cert_info.issuer ...is it X509_get_issuer_name(a) exactly the same?
>
> - a->cert_info.serialNumber.data ?
>
> - a->cert_info.serialNumber.length ?
>
> For completeness, my first, very raw code follows, where you can see 
> how I'd use the values.
>
> Thank you very much - Ubi
>
>
> #if OPENSSL_VERSION_NUMBER >= 0x30000000L
> #warning "I WILL HAVE MY LOCAL X509_issuer_and_serial_hash, UNDER 
> OPENSSL 3"
>
> unsigned long custom_X509_issuer_and_serial_hash(X509 *a)
> {
>     unsigned long ret = 0;
>     EVP_MD_CTX *ctx = EVP_MD_CTX_new();
>     unsigned char md[16];
>     char *f = NULL;
>
>     if (ctx == NULL)
>         goto err;
>         // cannot do this under OpenSSL 3 (code from v 1.1): f = 
> X509_NAME_oneline(a->cert_info.issuer, NULL, 0);
>         f = X509_NAME_oneline(X509_get_issuer_name(a), NULL, 0);
>     if (f == NULL)
>         goto err;
>     if (!EVP_DigestInit_ex(ctx, EVP_md5(), NULL))
>         goto err;
>     if (!EVP_DigestUpdate(ctx, (unsigned char *)f, strlen(f)))
>         goto err;
>     if (!EVP_DigestUpdate
>         // cannot do this under OpenSSL 3 (code from v 1.1): (ctx, 
> (unsigned char *)a->cert_info.serialNumber.data,
>         // ...but how do I get the data from here?
>         (ctx, X509_get_serialNumber(a),
>         // ...same problem here: how do I get the data length?
>         (unsigned long)a->cert_info.serialNumber.length))
>         goto err;
>     if (!EVP_DigestFinal_ex(ctx, &(md[0]), NULL))
>         goto err;
>     ret = (((unsigned long)md[0]) | ((unsigned long)md[1] << 8L) |
>            ((unsigned long)md[2] << 16L) | ((unsigned long)md[3] << 24L)
>         ) & 0xffffffffL;
>  err:
>     OPENSSL_free(f);
>     EVP_MD_CTX_free(ctx);
>     return ret;
> }
>
> #endif
>
>