EVP_PKEY_get_params strange behaviors

[ryan at splintermail.com]

I just migrated some JWK code from openssl 1.x to 3.x.

First I have to say, a lot of things got a lot easier and a lot clearer
than before.

I did see some strange behaviors with EVP_PKEY_get_params.  I see the
following statements from `man OSSL_PARAM`:

    [A] When requesting parameters, it's acceptable for data to be NULL.
    This can be used by the requestor to figure out dynamically exactly
    how much buffer space is needed to store the parameter data.  In
    this case, data_size is ignored.

    [B] If a responder finds that some data sizes are too small for the
    requested data, it must set return_size for each such OSSL_PARAM
    item to the minimum required size, and eventually return an error.

    [C] For the integer type parameters (OSSL_PARAM_UNSIGNED_INTEGER and
    OSSL_PARAM_INTEGER), a responder may choose to return an error if
    the data_size isn't a suitable size (even if data_size is bigger
    than needed).  If the responder finds the size suitable, it must
    fill all data_size bytes and ensure correct padding for the native
    endianness, and set return_size to the same value as data_size.

My understanding of [A] is that: when requesting parameters, if data is
NULL then data_size is *ignored*, meaning it does not affect any outputs
or behaviors of the function.

My understanding of [C] is that: it applies to each of the "qx", "qy",
and "priv" members of my EC-based pkey, which are all unsigned integers
according to `man EVP_PKEY-EC`.

But what I observe is that:

- [A] is not being respected, and EVP_PKEY_get_params() is returning
  errors when data_size is too small, even when data==NULL.

- In those failure cases, the return_size is not being set, which
  violates [B].

- When data is set and data_size is larger-than-necessary, qx and qy are
  behaving according to [C] but priv is not.


Reproducing code:

    #include <openssl/ec.h> #include <openssl/evp.h>

    int main(void){
        EVP_PKEY *pkey = EVP_EC_gen("P-256");
        int ret;

        #define UINT OSSL_PARAM_UNSIGNED_INTEGER
        OSSL_PARAM p1[] = {
            {.key="qx", .data_type=UINT, .data=NULL },
            {.key="qy", .data_type=UINT, .data=NULL },
            {.key="priv", .data_type=UINT, .data=NULL },
            {0},
        };
        ret = EVP_PKEY_get_params(pkey, p1);

        printf("ret = %d\n", ret);
        printf("xlen = %zu\n", p1[0].return_size);
        printf("ylen = %zu\n", p1[1].return_size);
        printf("dlen = %zu\n--\n", p1[2].return_size);

        // output:
        //   ret = 0      # Failure because of B, but only if you ignore A.
        //   xlen = 32
        //   ylen = 32
        //   dlen = 0     # Violates B.

        OSSL_PARAM p2[] = {
            {.key="qx", .data_type=UINT, .data=NULL, .data_size=SIZE_MAX },
            {.key="qy", .data_type=UINT, .data=NULL, .data_size=SIZE_MAX },
            {.key="priv", .data_type=UINT, .data=NULL, .data_size=SIZE_MAX },
            {0},
        };
        ret = EVP_PKEY_get_params(pkey, p2);

        printf("ret = %d\n", ret);
        printf("xlen = %zu\n", p2[0].return_size);
        printf("ylen = %zu\n", p2[1].return_size);
        printf("dlen = %zu\n--\n", p2[2].return_size);

        // output:     # What I wanted, but requires undocumented inputs.
        //   ret = 1
        //   xlen = 32
        //   ylen = 32
        //   dlen = 32

        char x[256];
        char y[256];
        char d[256];

        OSSL_PARAM p3[] = {
            {.key="qx", .data_type=UINT, .data=x, .data_size=256 },
            {.key="qy", .data_type=UINT, .data=y, .data_size=256 },
            {.key="priv", .data_type=UINT, .data=d, .data_size=256 },
            {0},
        };
        ret = EVP_PKEY_get_params(pkey, p3);

        printf("ret = %d\n", ret);
        printf("xlen = %zu\n", p3[0].return_size);
        printf("ylen = %zu\n", p3[1].return_size);
        printf("dlen = %zu\n", p3[2].return_size);

        // output:
        //   ret = 1
        //   xlen = 256  # Seems annoying, but this is what C. says.
        //   ylen = 256
        //   dlen = 32   # Seems helpful, but actually violates C.

        EVP_PKEY_free(pkey);

        return 0;
    }