Decrypt error when using openssl 1.1.1b during SSL handshake
Matt Caswell
matt at openssl.org
Mon Nov 2 11:19:17 UTC 2020
On 02/11/2020 09:00, Mahendra SP wrote:
> Hi Matt,
>
> Error is reported from this:
> FILE:../openssl-1.1.1b/ssl/statem/statem_srvr.c, FUNCTION:415,
> LINE:3055, reason=147, alert=51
>
> We see that hardware is returning 48 bytes. Even if the decrypted
> premaster data is correct, openssl is expecting more than 48 bytes in
> return.
> This check fails as decrypt_len is 48.
> decrypt_len < 11 + SSL_MAX_MASTER_KEY_LENGTH
Just above this line we call RSA_private_decrypt() with padding set to
RSA_NO_PADDING.
We expect the output *once padding is removed* to be 48 bytes. But
RSA_private_decrypt() should be returning the data *with the padding
included* (because we called it with RSA_NO_PADDING). The minimum valid
padding length is 11 bytes (hence the check above). So it looks to me
like the engine is incorrectly ignoring the RSA_NO_PADDING, and
stripping the padding anyway.
Matt
>
> We compared the data returned when software is used. Decrypt_len is 256
> bytes and the last 48 bytes are actual premaster secret. Also, openssl
> checks for if the first byte is 0 and second byte is 2.
> We are trying to rectify this issue in hardware and return the correct data.
>
> Please suggest if you have any comments for the above info.
>
> Thanks
> Mahendra
>
> On Fri, Oct 30, 2020 at 7:50 PM Matt Caswell <matt at openssl.org
> <mailto:matt at openssl.org>> wrote:
>
>
>
> On 30/10/2020 11:22, Mahendra SP wrote:
> > Hi Matt,
> >
> > Thank you for the inputs.
> > Yes, we had encountered the padding issue initially. But we added
> > support for RSA_NO_PADDING in our hardware. That's why we are able to
> > successfully decrypt the premaster secret in the hardware.
> > Hence the issue does not seem to be related to padding. We have
> > confirmed this by comparing the premaster secret on both client and
> > server and they are the same.
>
> Ok, good.
>
> >
> > We suspect in this case, verification of "encrypted handshake message"
> > failure is happening.
>
> It's possible. It would be helpful if you can get more information from
> the error stack on the server, e.g. by using ERR_print_errors_fp() or
> something similar. I'm particularly interested in identifying the source
> file and line number where the decrypt_error is coming from. Printing
> the error stack should give us that information. There are a number of
> places that a "decrypt error" can occur and it would be helpful to
> identify which one is the cause of the problem.
>
>
> > We understand constant_time_xx APIs get used for CBC padding
> validation.
> > Will this have any dependency on the compiler optimization or asm
> > flags?
>
> CBC padding validation is fairly independent of anything to do with RSA,
> so I think its unlikely to be the culprit here. Of course sometimes
> compiler optimization/asm issues do occur so it can't be ruled out
> entirely - but it's not where I would start looking.
>
> > Will this issue be seen if hardware takes more time for the
> > operation?
> >
>
> No. Constant time here just means that we implement the code without any
> branching based on secret data (e.g. no "if" statements/while loops etc
> based on secret dependent data). It has very little to do with how long
> something actually takes to process.
>
>
> > Here is the snippet of the wireshark where our device acting as server
> > closes the connection with decryption failure.
>
> Thanks. To narrow it down further I need to figure out which line of
> code the decrypt_error is coming from as described above.
>
> Matt
>
>
>
> > If you need any further info, please let us know.
> > image.png
> > Thanks
> > Mahendra
> >
> > Please suggest.
> >
> >
> >
> > On Fri, Oct 30, 2020 at 3:32 PM Matt Caswell <matt at openssl.org
> <mailto:matt at openssl.org>
> > <mailto:matt at openssl.org <mailto:matt at openssl.org>>> wrote:
> >
> >
> >
> > On 30/10/2020 09:18, Mahendra SP wrote:
> > > Hi All.
> > >
> > > We have upgraded openssl version to 1.1.1b
> > >
> > > With this, we are seeing decryption error during SSL
> handshake for the
> > > below explained scenario. Our device acts as an SSL server.
> > >
> > > We have external hardware to offload RSA private key
> operations using
> > > the engine.
> > > Decryption of pre-master secret is done using hardware and is
> > > successful. We compared the pre-master secret on both server and
> > client
> > > and they match.
> > > However, we see that SSL handshake fails with "decrypt error
> (51)"
> > with
> > > an alert number 21. Verifying the encrypted finish message
> on the
> > server
> > > side fails.
> > >
> > > This issue does not happen with software performing RSA
> private key
> > > operations.
> > >
> > > Can someone help with the reason for decryption failure?
> Below is the
> > > compiler and processor details. It is 64 bit.
> > > arm-linux-gnueabihf-gcc -march=armv7ve -mthumb -mfpu=neon
> > -mfloat-abi=hard
> >
> > Potentially this is related to the use of PSS padding in
> libssl which is
> > mandated in TLSv1.3. The TLSv1.3 spec also requires its use
> even in
> > TLSv1.2.
> >
> > The PSS padding is implemented within the EVP layer.
> Ultimately EVP
> > calls the function RSA_private_encrypt() with padding set to
> > RSA_NO_PADDING.
> >
> > Assuming your engine is implemented via a custom RSA_METHOD
> does it
> > support RSA_private_encrypt(() with RSA_NO_PADDING? If not this is
> > likely to be the problem.
> >
> > More discussion of this is here:
> >
> > https://github.com/openssl/openssl/issues/7968
> >
> > Also related is the recent discussion on this list about the
> CAPI engine
> > and this issue:
> >
> > https://github.com/openssl/openssl/issues/8872
> >
> > Matt
> >
>
More information about the openssl-users
mailing list