Endianness of EVP_PKEY_encryp output

[Viktor Dukhovni]

On Tue, Apr 04, 2023 at 11:02:33PM +0000, Viken Kondakji wrote:

> I am using EVP_PKEY_encrypt, and running on a little-endian host using
> RSA public key.  Will the binary output of my encryption be in
> big-endian?

Your mistake is using EVP_PKEY_encrypt() directly, rather than a high
level primitive, like CMS_EncryptedData_encrypt().

> The target destination wants the binary data in big-endian and then
> encoded to base 64.

If they are looking for raw RSA signatures, they too are misguided.

> A man of EVP_PKEY_encrypt(3) on my host does not mention any
> endianness.

You shouldn't be using this function directly.  It is only for use in
higher-level protocols.  Naive misuse inevitably leads to significant
security issues.

-- 
    Viktor.