[openssl-users] "digest check failure" with AmigaOS3/m68k port of OpenSSL 1.0.x

[Jens Maus]

Hello,

I am the current maintainer of a still active port of OpenSSL to the AmigaOS platform which tries to wrap the OpenSSL library API into a full fledged Amiga shared library for applications requiring cryptographic functionality (see https://github.com/jens-maus/amissl). So yes, the Amiga platform is still alive ;)

While for some Amiga platforms (e.g. AmigaOS4/PPC) the current OpenSSL 1.0.2f kernel of this library seems to behave fine and all our tests are not reporting any problem we are still facing some trouble with one of the older Amiga platforms (AmigaOS3) which utilizes Motorola m68k processors. While all of the openssl test binaries are not outputting any error, we are facing some trouble in receiving „digest check failed“ messages, e.g. when executing the following ‚openssl‘ test command:

openssl s_client -connect pop.gmail.com:995 -tls1_2 -cipher ECDHE-RSA-AES128-GCM-SHA256

The problem vanishes, however, immediately when using a SHA384 using cipher:

openssl s_client -connect pop.gmail.com:995 -tls1_2 -cipher ECDHE-RSA-AES256-GCM-SHA384

The error output we are receiving when using SHA256 digest ciphers is:

error:1408C095:SSL routines:ssl3_get_finished:digest check failed

Please note, however, that the „sha256t“ openssl test programs doesn’t output any error nor does a „openssl dgst -sha256“ command produce any broken SHA256 digest outputs.

After having tracked down the problem in the OpenSSL source code we have traced down the problem to the following CRYPTO_memcmp() failing for some unknown reason:

https://github.com/openssl/openssl/blob/OpenSSL_1_0_2f/ssl/s3_both.c#L271

So in this case either s->init_msg or s->s3->tmp.peer_finish_md seems to be incorrectly calculated. Commenting out the whole CRYPTO_memcmp() check results, however, in a succeeding TLS connection where s_client can then properly communicate with the server in question.

Our current difficulty in trying to debug if either init_msg or peer_finish_md is incorrectly calculated is, that the corresponding code passages are of course using random values and thus each connection produces differences we can hardly compare to each other.

I would like to therefore ask if there is any possibility or defined way of debugging/analyzing TLS connection handshakes with the exact same handshake procedure so that successive uses of „openssl s_client“ will always produce the same output? Or how do I have to manually calculate the SHA256 digest based on the TLS handshake data I am receiving via „openssl s_client -msg“ output? In addition, I would like to ask if anyone has another idea how I could debug why the SHA256 digest seems to be incorrectly calculated when performing a TLS1.2 connection?!?

If anyone is interested, here is the corresponding github ticket which we are maintaining to track down the problem:
https://github.com/jens-maus/amissl/issues/2

Any help of course very appreciated!

Best Regards,
Jens
-- 
Jens Maus, Dresden/Germany
http://jens-maus.de/

*** Content is authentic only with digital signature  ***

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 2605 bytes
Desc: not available
URL: <http://mta.openssl.org/pipermail/openssl-users/attachments/20160225/5d7ad225/attachment.bin>