Decrypt error when using openssl 1.1.1b during SSL handshake

[Matt Caswell]

On 30/10/2020 11:22, Mahendra SP wrote:
> Hi Matt,
> 
> Thank you for the inputs.
> Yes, we had encountered the padding issue initially. But we added
> support for RSA_NO_PADDING in our hardware. That's why we are able to
> successfully decrypt the premaster secret in the hardware.
> Hence the issue does not seem to be related to padding. We have
> confirmed this by comparing the premaster secret on both client and
> server and they are the same.

Ok, good.

> 
> We suspect in this case, verification of "encrypted handshake message"
> failure is happening.

It's possible. It would be helpful if you can get more information from
the error stack on the server, e.g. by using ERR_print_errors_fp() or
something similar. I'm particularly interested in identifying the source
file and line number where the decrypt_error is coming from. Printing
the error stack should give us that information. There are a number of
places that a "decrypt error" can occur and it would be helpful to
identify which one is the cause of the problem.


> We understand constant_time_xx APIs get used for CBC padding validation.
> Will this have  any dependency on the compiler optimization or asm
> flags?

CBC padding validation is fairly independent of anything to do with RSA,
so I think its unlikely to be the culprit here. Of course sometimes
compiler optimization/asm issues do occur so it can't be ruled out
entirely - but it's not where I would start looking.

> Will this issue be seen if hardware takes more time for the
> operation?
> 

No. Constant time here just means that we implement the code without any
branching based on secret data (e.g. no "if" statements/while loops etc
based on secret dependent data). It has very little to do with how long
something actually takes to process.


> Here is the snippet of the wireshark where our device acting as server
> closes the connection with decryption failure.

Thanks. To narrow it down further I need to figure out which line of
code the decrypt_error is coming from as described above.

Matt



> If you need any further info, please let us know. 
> image.png
> Thanks
> Mahendra
> 
> Please suggest.
> 
> 
> 
> On Fri, Oct 30, 2020 at 3:32 PM Matt Caswell <matt at openssl.org
> <mailto:matt at openssl.org>> wrote:
> 
> 
> 
>     On 30/10/2020 09:18, Mahendra SP wrote:
>     > Hi All.
>     >
>     > We have upgraded openssl version to 1.1.1b
>     >
>     > With this, we are seeing decryption error during SSL handshake for the
>     > below explained scenario. Our device acts as an SSL server.
>     >
>     > We have external hardware to offload RSA private key operations using
>     > the engine.
>     > Decryption of pre-master secret is done using hardware and is
>     > successful. We compared the pre-master secret on both server and
>     client
>     > and they match.
>     > However, we see that SSL handshake fails with "decrypt error (51)"
>     with
>     > an alert number 21. Verifying the encrypted finish message on the
>     server
>     > side fails.
>     >
>     > This issue does not happen with software performing RSA private key
>     > operations.
>     >
>     > Can someone help with the reason for decryption failure? Below is the
>     > compiler and processor details. It is 64 bit.
>     > arm-linux-gnueabihf-gcc  -march=armv7ve -mthumb -mfpu=neon
>     -mfloat-abi=hard
> 
>     Potentially this is related to the use of PSS padding in libssl which is
>     mandated in TLSv1.3. The TLSv1.3 spec also requires its use even in
>     TLSv1.2.
> 
>     The PSS padding is implemented within the EVP layer. Ultimately EVP
>     calls the function RSA_private_encrypt() with padding set to
>     RSA_NO_PADDING.
> 
>     Assuming your engine is implemented via a custom RSA_METHOD does it
>     support RSA_private_encrypt(() with RSA_NO_PADDING? If not this is
>     likely to be the problem.
> 
>     More discussion of this is here:
> 
>     https://github.com/openssl/openssl/issues/7968
> 
>     Also related is the recent discussion on this list about the CAPI engine
>     and this issue:
> 
>     https://github.com/openssl/openssl/issues/8872
> 
>     Matt
>