decoder memory-management question

[D. J. Bernstein]

To give a bit more context: The attached beta openssl_ed25519_lib25519.c
is an OpenSSL provider that uses lib25519 to speed up Ed25519. On, e.g.,
a 3.5GHz Haswell core, openssl speed with OpenSSL's built-in provider
says 21239.0 sign/s and 8272.0 verify/s, while openssl speed with this
provider says 73708.1 sign/s and 22853.0 verify/s. I used a different
secret-key format (and maybe cheated a bit on the OIDs for that) but the
public keys are compatible.

I'm planning similar X25519 integration. For an idea of the speedups on
various Intel/AMD platforms, see https://lib25519.cr.yp.to/speed.html.
There's also ARM code coming soon. An important caveat is that the code
isn't formally verified yet; but there's some fully verified code from
s2n-bignum that drops straightforwardly into lib25519, and reusing that
code helps security even for applications that don't care about speed.

I haven't dug this deeply into OpenSSL before. I could easily be doing
major things wrong in this provider, maybe security-relevant things,
perhaps most importantly regarding memory management. The documentation
that I looked at had occasional comments on who was responsible for each
piece of memory at each moment, but most of the time I found myself
running tests and studying code samples (in the OpenSSL core, OpenSSL's
default provider, and a few providers I found online) to try to figure
out the expectations.

Some memory-management issues that I bumped into were easily avoidable.
For example, bad memory management in the OpenSSL core, specifically a
bad free() for openssl req, occurs if a provider returns bad _bytes_
inside an OSSL_SIGNATURE_PARAM_ALGORITHM_ID octet string. Well, okay,
obviously I should get the ID right. (The core should be fixed too:
string contents should never be able to corrupt pointer structures.)

As another example, I think I understand the layering that's expected to
be used inside encoders and decoders, but the layers for public-key
decoding were turning into such a data-structure mess that I ended up
just tossing that code and doing a trivial parse of the known DER prefix
followed by the Ed25519 byte string. It's tempting to simplify the other
encoding/decoding steps in the same way.

But obviously a provider can't avoid some issues. In particular, there's
a three-way dance between decode() and its callback and load(), and I
want to make sure that I'm handling memory in the expected way for all
possible failure cases. It would be nice if OpenSSL included a test
harness for providers---not just basic functionality tests as in the
attached test.sh but also testing how various types of internal failures
are handled.

---D. J. Bernstein
-------------- next part --------------
A non-text attachment was scrubbed...
Name: openssl_ed25519_lib25519.c
Type: text/x-csrc
Size: 24574 bytes
Desc: not available
URL: <https://mta.openssl.org/pipermail/openssl-users/attachments/20240302/c60d705d/attachment-0001.c>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: test.sh
Type: application/x-sh
Size: 5967 bytes
Desc: not available
URL: <https://mta.openssl.org/pipermail/openssl-users/attachments/20240302/c60d705d/attachment-0001.sh>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <https://mta.openssl.org/pipermail/openssl-users/attachments/20240302/c60d705d/attachment-0001.sig>