[openssl-dev] Upgrading OpenSSL

[Dr. Stephen Henson]

On Mon, Aug 28, 2017, Leon Brits wrote:

> The upgrade is now working fine in one of the applications which make TLS connections. I can see the engine functions being called when some action (sign/verify) which require the privatekey.
> 
> However, this engine is also used in a patched version of Racoon2.
> In one of the files (crypto_openssl.c) a function is called by the IKE daemon (iked) during setup with:
>             :
> EVP_SignInit(&ctx, md);
> EVP_SignUpdate(&ctx, octets->v, octets->l);
> EVP_SignFinal(&ctx, (unsigned char*)sig->v, &siglen, pkey);
> :
> With the upgraded OpenSSL v1.0.2, the last function now fails with the error:
> 2017-08-28 15:44:14 [INTERNAL_ERR]: crypto_openssl.c:1238:eay_rsassa_pkcs1_v1_5_sign(): RSA_sign failed: 30972:error:0606B06E:digital envelope routines:EVP_SignFinal:wrong public key type:p_sign.c:123:
> 
> Q: I assume it is looking for one of the missing parameters (p) in the RSA structure - correct?
> Q: If so, how did it work in v0.9.8?
> 
> If I change the first command to:
> EVP_SignInit_ex(&ctx, md, engine);
> Then it segfaults in the "SignFinal" command :(
> 
> The engine is dynamically loaded the same way in both my TLS connection application and in the Racoon2 application.
> 
> Thanks for your time - any help is appreciated!
> 

Have you recompiled the application? Some structures have changed between
OpenSSL 0.9.8 and 1.0.2 so you'll get problems with applications (or an
ENGINE) compiled against the wrong headers.

If that isn't the problem then what is "md" set to?

Steve.
--
Dr Stephen N. Henson. OpenSSL project core developer.
Commercial tech support now available see: http://www.openssl.org