Decrypt error when using openssl 1.1.1b during SSL handshake

[Matt Caswell]

On 30/10/2020 09:18, Mahendra SP wrote:
> Hi All.
> 
> We have upgraded openssl version to 1.1.1b
> 
> With this, we are seeing decryption error during SSL handshake for the
> below explained scenario. Our device acts as an SSL server.
> 
> We have external hardware to offload RSA private key operations using
> the engine.
> Decryption of pre-master secret is done using hardware and is
> successful. We compared the pre-master secret on both server and client
> and they match.
> However, we see that SSL handshake fails with "decrypt error (51)" with
> an alert number 21. Verifying the encrypted finish message on the server
> side fails.
> 
> This issue does not happen with software performing RSA private key
> operations.
> 
> Can someone help with the reason for decryption failure? Below is the
> compiler and processor details. It is 64 bit.
> arm-linux-gnueabihf-gcc  -march=armv7ve -mthumb -mfpu=neon -mfloat-abi=hard

Potentially this is related to the use of PSS padding in libssl which is
mandated in TLSv1.3. The TLSv1.3 spec also requires its use even in TLSv1.2.

The PSS padding is implemented within the EVP layer. Ultimately EVP
calls the function RSA_private_encrypt() with padding set to RSA_NO_PADDING.

Assuming your engine is implemented via a custom RSA_METHOD does it
support RSA_private_encrypt(() with RSA_NO_PADDING? If not this is
likely to be the problem.

More discussion of this is here:

https://github.com/openssl/openssl/issues/7968

Also related is the recent discussion on this list about the CAPI engine
and this issue:

https://github.com/openssl/openssl/issues/8872

Matt