[openssl-users] ECDSA digest configurations

[Dr. Stephen Henson]

On Fri, Apr 24, 2015, Rajeswari K wrote:

> Hello openssl-users,
> 
> I have an issue with update of ECDSA digests in our environment.
> 
> We have our own digest functions for init, update and final where we
> registered with these functions for NID_sha1, NID_sha256, NID_sha384 and
> NID_sha512. These digests were updated at openssl, via ENGINE_set_digests()
> function.
> 
> After update, i see that all RSA cerificate based ciphers are working fine
> with both TLS1.0 and TLS1.2.
> 
> And, i added the ECDSA algorithm with openssl function
> EVP_add_digest(EVP_ecdsa());
> 
> But, for me ECDSA cert based cipher suites are not working. These are
> landing to following errors.
> 
> digital envelope routines:EVP_SignFinal:wrong public key type :
> crypto/evp/p_sign     .c:139:
> 
> SSL routines:SSL3_SEND_SERVER_KEY_EXCHANGE:EVP lib: ssl/s3_srvr.c:2012
> 
> 
> Then i tried following
> Tried to set only NID_ecdsa_with_SHA1 with our init, update and final
> function and tried to update to openssl via ENGINE_set_digests().
> 
> But, now seems to be since i updated NID_ecdsa_with_SHA1 with our sha1
> functions, whole TLS1.0 handshakes stopped working with "decrypt error or
> bac mac record" errors.
> 
> Based on this, seems to be there is a basic thing am missing while
> configuring ECDSA based digests.
> 
> Can you please help me on this configuration? How can we differentiate
> between ECDSA digests to RSA digests?
> 
> Because both usually lands on to same SHA1, SHA256 etc functions.
> 
> I tried one more thing that, along with NID_sha1, NID_sha256, NID_sha384,
> NID_sha512 i tried to define NID_ecdsa_with_SHA1, NID_ecdsa_with_SHA256,
> NID_ecdsa_with_SHA384 and NID_ecdsa_with_SHA512 with respective settings.
> But, with this setting also landing to same above error of "wrong public
> key type".
> 
> I need your inputs on this issue.
> 

So does your code just perform the digest operation and not signing?

Which version was it working with before?

There isn't really any difference between an ECDSA and RSA digest. Older
versions of OpenSSL linked digests and signing algorithms. That link no longer
exists and some algorithms are marked as being suitable for multiple public
key algorithms.

Are you setting the flag EVP_MD_FLAG_PKEY_METHOD_SIGNATURE in the EVP_MD
structure?

An ENGINE providing alternative implementations of built in digests shoudn't
need to call EVP_add_digest. If everything is set up properly OpenSSL should
just switch to the ENGINE implementation.

Steve.
--
Dr Stephen N. Henson. OpenSSL project core developer.
Commercial tech support now available see: http://www.openssl.org