Endianness of EVP_PKEY_encryp output

[Viktor Dukhovni]

On Wed, Apr 05, 2023 at 12:34:09PM +0000, Viken Kondakji wrote:
> Re my question about endianness, I am not in control.
> I am required to do what the target destination tells me to do.  It is that simple.
> 
> So, I ask again:
> Will the binary output of the EVP_PKEY_encrypt on a small-endian host produce binary data in big-endian?
> Can anyone help, please.

The output is a bytestring not an integer, so the endianness question is
not applicable.  That said, the serialisation of primitive RSA
encryption operation results is "big-endian" per various specifications.

You still should not use (or at least treat with great scepticisim) any
protocol where you're performing raw RSA encryption operations.

-- 
    Viktor.