[openssl-users] EVP_DecryptUpdate: why is this failing when out == in?

[Paul Smith]

I filed https://github.com/openssl/openssl/issues/7941 about this FYI.

Cheers!


On Wed, 2018-12-19 at 01:56 -0500, Paul Smith wrote:
> As I understand it, it's legal to provide the exact same input and
> output buffer to EVP_EncryptUpdate and EVP_DecryptUpdate, but it's not
> legal to provide pointers into different parts of the same buffer. 
> That's a good check.
> 
> However, my implementation is getting triggered by this code in
> EVP_DecryptUpdate():
> 
>     if (ctx->final_used) {
>         /* see comment about PTRDIFF_T comparison above */
> =>      if (((PTRDIFF_T)out == (PTRDIFF_T)in)
>             || is_partially_overlapping(out, in, b)) {
>             EVPerr(EVP_F_EVP_DECRYPTUPDATE, EVP_R_PARTIALLY_OVERLAPPING);
>             return 0;
>         }
> 
> Can someone explain why, only in this specific situation where we're
> decrypting the final block, we require that OUT and IN not be the same
> buffer?  Everywhere else we check is_partially_overlapping() only,
> without equality.
> 
> I read the comment about PTRDIFF_T but I didn't come up with a reason
> for the equality check.  This check was added back in 2016 in SHA
> 5fc77684f1 FWIW.