OpenSSL 3.0 password prompt errors

[Matt Caswell]

On 30/11/2021 13:16, pepone.onrez wrote:
> Getting some problems with OpenSSL 3.0, I have passwordError function, 
> to check if the last error was due to an invalid password and allow the 
> user to retry.
> 
> 
> bool
> passwordError()
> {
>      unsigned long error = ERR_peek_error();
>      unsigned long lib = ERR_GET_LIB(error);
>      unsigned long reason = ERR_GET_REASON(error);
>      cerr << "error: " << error << endl;
>      cerr << "lib: " << lib << endl;
>      cerr << "reason: " << reason << endl;
>      ERR_print_errors_fp(stdout);
>      return (reason == PEM_R_BAD_BASE64_DECODE ||
>              reason == PEM_R_BAD_DECRYPT ||
>              reason == PEM_R_BAD_PASSWORD_READ ||
>              reason == PEM_R_PROBLEMS_GETTING_PASSWORD ||
>              reason == PKCS12_R_MAC_VERIFY_FAILURE);
> }
> 
> When I test with an invalid password I get
> 
> error: 587686001
> lib: 70
> reason: 483441
> error:23076071:PKCS12 routines:PKCS12_parse:mac verify failure

That is really screwy output. Something is getting corrupted somewhere. 
"70" is not a valid error library and a reason code of 483441 is clearly 
wrong (reason codes are typically fairly small). Error 587686001 does 
correspond to the hex value 23076071 - but this is not an error value I 
would expect to see OpenSSL emitting.

Could there be memory corruption occurring?? Perhaps run this through 
valgrind or similar and see if there are any hints.

Matt



> 
> 
> the description seems to match PKCS12_R_MAC_VERIFY_FAILURE but the 
> reason value doesn't
> 
> include/openssl/pkcs12err.h
> 39:# define PKCS12_R_MAC_VERIFY_FAILURE                      113
> 
> Any ideas what I might be doing wrong here? this worked fine with 1.1.1 
> before
> 
> Cheers,
> Jose