Decrypt error when using openssl 1.1.1b during SSL handshake

[Mahendra SP]

Hi Matt,

Error is reported from this:
FILE:../openssl-1.1.1b/ssl/statem/statem_srvr.c, FUNCTION:415, LINE:3055,
reason=147, alert=51

We see that hardware is returning 48 bytes. Even if the decrypted premaster
data is correct, openssl is expecting more than 48 bytes in return.
This check fails as decrypt_len is 48.
decrypt_len < 11 + SSL_MAX_MASTER_KEY_LENGTH

We compared the data returned when software is used. Decrypt_len is 256
bytes and the last 48 bytes are actual premaster secret. Also, openssl
checks for if the first byte is 0 and second byte is 2.
We are trying to rectify this issue in hardware and return the correct data.

Please suggest if you have any comments for the above info.

Thanks
Mahendra

On Fri, Oct 30, 2020 at 7:50 PM Matt Caswell <matt at openssl.org> wrote:

>
>
> On 30/10/2020 11:22, Mahendra SP wrote:
> > Hi Matt,
> >
> > Thank you for the inputs.
> > Yes, we had encountered the padding issue initially. But we added
> > support for RSA_NO_PADDING in our hardware. That's why we are able to
> > successfully decrypt the premaster secret in the hardware.
> > Hence the issue does not seem to be related to padding. We have
> > confirmed this by comparing the premaster secret on both client and
> > server and they are the same.
>
> Ok, good.
>
> >
> > We suspect in this case, verification of "encrypted handshake message"
> > failure is happening.
>
> It's possible. It would be helpful if you can get more information from
> the error stack on the server, e.g. by using ERR_print_errors_fp() or
> something similar. I'm particularly interested in identifying the source
> file and line number where the decrypt_error is coming from. Printing
> the error stack should give us that information. There are a number of
> places that a "decrypt error" can occur and it would be helpful to
> identify which one is the cause of the problem.
>
>
> > We understand constant_time_xx APIs get used for CBC padding validation.
> > Will this have  any dependency on the compiler optimization or asm
> > flags?
>
> CBC padding validation is fairly independent of anything to do with RSA,
> so I think its unlikely to be the culprit here. Of course sometimes
> compiler optimization/asm issues do occur so it can't be ruled out
> entirely - but it's not where I would start looking.
>
> > Will this issue be seen if hardware takes more time for the
> > operation?
> >
>
> No. Constant time here just means that we implement the code without any
> branching based on secret data (e.g. no "if" statements/while loops etc
> based on secret dependent data). It has very little to do with how long
> something actually takes to process.
>
>
> > Here is the snippet of the wireshark where our device acting as server
> > closes the connection with decryption failure.
>
> Thanks. To narrow it down further I need to figure out which line of
> code the decrypt_error is coming from as described above.
>
> Matt
>
>
>
> > If you need any further info, please let us know.
> > image.png
> > Thanks
> > Mahendra
> >
> > Please suggest.
> >
> >
> >
> > On Fri, Oct 30, 2020 at 3:32 PM Matt Caswell <matt at openssl.org
> > <mailto:matt at openssl.org>> wrote:
> >
> >
> >
> >     On 30/10/2020 09:18, Mahendra SP wrote:
> >     > Hi All.
> >     >
> >     > We have upgraded openssl version to 1.1.1b
> >     >
> >     > With this, we are seeing decryption error during SSL handshake for
> the
> >     > below explained scenario. Our device acts as an SSL server.
> >     >
> >     > We have external hardware to offload RSA private key operations
> using
> >     > the engine.
> >     > Decryption of pre-master secret is done using hardware and is
> >     > successful. We compared the pre-master secret on both server and
> >     client
> >     > and they match.
> >     > However, we see that SSL handshake fails with "decrypt error (51)"
> >     with
> >     > an alert number 21. Verifying the encrypted finish message on the
> >     server
> >     > side fails.
> >     >
> >     > This issue does not happen with software performing RSA private key
> >     > operations.
> >     >
> >     > Can someone help with the reason for decryption failure? Below is
> the
> >     > compiler and processor details. It is 64 bit.
> >     > arm-linux-gnueabihf-gcc  -march=armv7ve -mthumb -mfpu=neon
> >     -mfloat-abi=hard
> >
> >     Potentially this is related to the use of PSS padding in libssl
> which is
> >     mandated in TLSv1.3. The TLSv1.3 spec also requires its use even in
> >     TLSv1.2.
> >
> >     The PSS padding is implemented within the EVP layer. Ultimately EVP
> >     calls the function RSA_private_encrypt() with padding set to
> >     RSA_NO_PADDING.
> >
> >     Assuming your engine is implemented via a custom RSA_METHOD does it
> >     support RSA_private_encrypt(() with RSA_NO_PADDING? If not this is
> >     likely to be the problem.
> >
> >     More discussion of this is here:
> >
> >     https://github.com/openssl/openssl/issues/7968
> >
> >     Also related is the recent discussion on this list about the CAPI
> engine
> >     and this issue:
> >
> >     https://github.com/openssl/openssl/issues/8872
> >
> >     Matt
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mta.openssl.org/pipermail/openssl-users/attachments/20201102/1a373723/attachment.html>