[openssl-dev] Fwd: latest OpenSSL causes OpenSMTPD to segv

[Richard Levitte]

In message <20160201231650.GF4987 at mournblade.imrryr.org> on Mon, 1 Feb 2016 23:16:50 +0000, Viktor Dukhovni <openssl-users at dukhovni.org> said:

openssl-users> On Mon, Feb 01, 2016 at 10:52:56PM +0000, Viktor Dukhovni wrote:
openssl-users> 
openssl-users> > The only thing I see that's plausibly pertinent is:
openssl-users> > 
openssl-users> > commit 6656ba7152dfe4bba865e327dd362ea08544aa80
openssl-users> > Author: Dr. Stephen Henson <steve at openssl.org>
openssl-users> > Date:   Sun Dec 20 18:18:43 2015 +0000
openssl-users> > 
openssl-users> >     Don't check RSA_FLAG_SIGN_VER.
openssl-users> > 
openssl-users> >     Reviewed-by: Richard Levitte <levitte at openssl.org>
openssl-users> > 
openssl-users> 
openssl-users> This is related to:
openssl-users> 
openssl-users>     commit 1c80019a2c8f59410552197723829fd72ab45a5e
openssl-users>     Author: Dr. Stephen Henson <steve at openssl.org>
openssl-users>     Date:   Sat Sep 18 22:37:44 1999 +0000
openssl-users> 
openssl-users> 	Add new sign and verify members to RSA_METHOD and change SSL code to use sign
openssl-users> 	and verify rather than direct encrypt/decrypt.
openssl-users> 
openssl-users> Which was already present in 0.9.7.  Thus, presumably engines have
openssl-users> been expected to implement the "new" methods, if they were ported
openssl-users> to OpenSSL 0.9.7 or later.
openssl-users> 
openssl-users> It seems that perhaps the need to implemnt sign/verify and not just
openssl-users> encrypt/decrypt has not been communicated to the engine maintainers.
openssl-users> 
openssl-users> The master branch has:
openssl-users> 
openssl-users>     commit 19c6d3ea2d3b4e0ad3e978e42cc7cbdf0c09891f
openssl-users>     Author: Dr. Stephen Henson <steve at openssl.org>
openssl-users>     Date:   Wed Dec 2 14:30:39 2015 +0000
openssl-users> 
openssl-users> 	Remove RSA_FLAG_SIGN_VER flag.
openssl-users> 
openssl-users> 	Remove RSA_FLAG_SIGN_VER: this was origininally used to retain binary
openssl-users> 	compatibility after RSA_METHOD was extended to include rsa_sign and
openssl-users> 	rsa_verify fields. It is no longer needed.
openssl-users> 
openssl-users> 	Reviewed-by: Richard Levitte <levitte at openssl.org>
openssl-users> 
openssl-users> And while indeed the structure has been stable with sign/verify
openssl-users> methods for ages, engines that don't implement sign/verify may well
openssl-users> exist, so dropping the flag check can break some engines.

Hold on a minute...  there is a test that the function pointer is
assigned:

    if (rsa->meth->rsa_sign) {
        return rsa->meth->rsa_sign(type, m, m_len, sigret, siglen, rsa);
    }

So what I can conclude without looking is that one of two things have
happened:

1. the RSA_METHOD hasn't been fully initialised, so the rsa_sign
   pointer is garbage.

2. the function that rsa_sign points as is faulty in some way, but has
   never been called before now because there was no RSA_FLAG_SIGN_VER
   bit present.

I just downloaded the latest portable OpenSMTPD and am noticing that
rsa_sign, rsa_verify and rsa_keygen are filled in (with rsae_sign,
rsae_verify and rsae_keygen), but that there are no bits at all
assigned to the flags field.  As far as I can see, this means that
these functions have never been called...  before now.

Ref: opensmtpd-5.7.3p1.tar.gz, smtpd/ca.c

Cheers,
Richard

-- 
Richard Levitte         levitte at openssl.org
OpenSSL Project         http://www.openssl.org/~levitte/