Decrypt error when using openssl 1.1.1b during SSL handshake

[Mahendra SP]

Hi Matt,

Thank you for the inputs.
Yes, we had encountered the padding issue initially. But we added support
for RSA_NO_PADDING in our hardware. That's why we are able to successfully
decrypt the premaster secret in the hardware.
Hence the issue does not seem to be related to padding. We have confirmed
this by comparing the premaster secret on both client and server and they
are the same.

We suspect in this case, verification of "encrypted handshake message"
failure is happening.
We understand constant_time_xx APIs get used for CBC padding validation.
Will this have  any dependency on the compiler optimization or asm flags?
Will this issue be seen if hardware takes more time for the operation?

Here is the snippet of the wireshark where our device acting as server
closes the connection with decryption failure.
If you need any further info, please let us know.
[image: image.png]
Thanks
Mahendra

Please suggest.



On Fri, Oct 30, 2020 at 3:32 PM Matt Caswell <matt at openssl.org> wrote:

>
>
> On 30/10/2020 09:18, Mahendra SP wrote:
> > Hi All.
> >
> > We have upgraded openssl version to 1.1.1b
> >
> > With this, we are seeing decryption error during SSL handshake for the
> > below explained scenario. Our device acts as an SSL server.
> >
> > We have external hardware to offload RSA private key operations using
> > the engine.
> > Decryption of pre-master secret is done using hardware and is
> > successful. We compared the pre-master secret on both server and client
> > and they match.
> > However, we see that SSL handshake fails with "decrypt error (51)" with
> > an alert number 21. Verifying the encrypted finish message on the server
> > side fails.
> >
> > This issue does not happen with software performing RSA private key
> > operations.
> >
> > Can someone help with the reason for decryption failure? Below is the
> > compiler and processor details. It is 64 bit.
> > arm-linux-gnueabihf-gcc  -march=armv7ve -mthumb -mfpu=neon
> -mfloat-abi=hard
>
> Potentially this is related to the use of PSS padding in libssl which is
> mandated in TLSv1.3. The TLSv1.3 spec also requires its use even in
> TLSv1.2.
>
> The PSS padding is implemented within the EVP layer. Ultimately EVP
> calls the function RSA_private_encrypt() with padding set to
> RSA_NO_PADDING.
>
> Assuming your engine is implemented via a custom RSA_METHOD does it
> support RSA_private_encrypt(() with RSA_NO_PADDING? If not this is
> likely to be the problem.
>
> More discussion of this is here:
>
> https://github.com/openssl/openssl/issues/7968
>
> Also related is the recent discussion on this list about the CAPI engine
> and this issue:
>
> https://github.com/openssl/openssl/issues/8872
>
> Matt
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mta.openssl.org/pipermail/openssl-users/attachments/20201030/4fa768c7/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 24038 bytes
Desc: not available
URL: <https://mta.openssl.org/pipermail/openssl-users/attachments/20201030/4fa768c7/attachment-0001.png>