[openssl-users] Authentication over ECDHE

[Matt Caswell]

On 31/12/2018 11:36, C.Wehrmeyer wrote:
> On 31.12.18 10:12, Richard Levitte wrote:
>> Yes, it's true, new features are going in.  And it's true that it's
>> often more exciting to add new features than to do the janitorial
>> work.
> 
> You realised what I have left unspoken thus far, which is this almost
> obsession-like preference of OSS coders to add new features rather than
> improving the old, boring codebase. However, there's a reason why it's still
> called code*base*. It's the base of everything. And adding more and more
> features to that base is going to make ripping off the band-aid more painful in
> the long run.

There has been a huge amount of effort put in over the last few years to improve
the codebase. Things that immediately spring to mind (and there's probably a
whole load more):

- Rewrite of the state machine
- libssl record layer refactor
- Implementation of the PACKET and WPACKET abstractions in libssl
- Rewrite of the rand code
- Implementation of the new test harness
- Significant effort into developing tests
- Implementation of the coding style and reformat of the codebase to meet it
- Opaque many of the structures (which I know you don't see as an improvement,
but I'll answer that point separately)
- Implementation of continuous fuzzing
- Significant expansion of the documentation coverage

It is simply not true to claim that we have "an obsession-like preference...to
add new features rather than improving the old, boring codebase". None of the
above things resulted in or were motivated by user visible features. They were
all about improving the codebase.

> 
> Also, infrastructure again. I, as a user, don't care if the kernel gets a new
> feature that makes some black magic happening. What I care about is that the
> kernel doesn't throw away my writes (which has happened in May of 2018, see):
> 
>>
> https://www.postgresql.org/message-id/flat/CAMsr%2BYE5Gs9iPqw2mQ6OHt1aC5Qk5EuBFCyG%2BvzHun1EqMxyQg%40mail.gmail.com#CAMsr+YE5Gs9iPqw2mQ6OHt1aC5Qk5EuBFCyG+vzHun1EqMxyQg@mail.gmail.com)
> 
> 
> Cryptography libs should be equally conservative, considering that cryptography
> is conservative to begin with. I don't care if TLS 1.3 lets me use new exiting
> ciphers and handshakes when it unreasonably bogs down my server code.
> 
>> BUT, you also have to appreciate that stuff is happening around us
>> that affects our focus.  TLS 1.3 happened, and rather than having to
>> answer the question "why don't you have TLS 1.3 yet?" (there's a LOT
>> of interest in that version), we decided to add it.
> 
> Sure, but didn't Matt just say that there are a lot of volunteers working on
> that library? The disadvantage here is that quality assurance is barely a thing
> - however, the *advantage* of this is that OpenSSL does not have to follow
> commercial interests. If we look at this at face value you could just say "No,
> people, it's high time we streamline some of the internal aspects of the
> library, TLS 1.3 will have to wait. You can't wait that long? Well, sorry".
> 
>> However, your message is clear, we do need to do some cleanup as
>> well.  More than that, I agree with you that it's needed (I've
>> screamed out in angst when stumbling upon particularly ugly or
>> misplaced code, so the feeling is shared, more than you might
>> believe).
> 
> But what does "cleanup" entail? That's the hot-button question here. I've
> already made a suggestion, that is to say, getting rid of opaque structures. If
> that is deemed too insecure (for whatever reasons), export symbols that allow
> programmers to query the size of structures, and provide two versions of
> functions: one function expects the caller to pass an object to which the
> changes are to be made, and the other one allocates the necessary memory
> dynamically and then calls the first version. Or just don't allocate my object
> memory dynamically anymore.
> 
>> That being said, cleanup happens, and documentation happens, in a
>> piecemeal fashion, 'cause that's what most people have capacity for.
> 
> So, what you're effectively saying is that I'm the first one who ever asked for
> SSL object reuse, right? Because if piecemeal work happens on the documentation,
> and Viktor says that it's possible, then surely no one would have ever answered
> that question on the mailing list and *not* put it piecemeal-ly in the OpenSSL
> documentation, right?
> 
>> Now, here's something else that you need to consider: API/ABI
>> compatibility needs to be preserved.
> 
> No it doesn't. We *know* it doesn't. When OpenSSL 1.1 was released it broke all
> *sorts* of applications out there, because all sorts of applications used struct
> fields rather than accessors. wget, mutt, neon, python, you name it, you broke it.

API/ABI stability is absolutely required. Every time we make a breaking change
it is painful for our users - and more pain is felt the bigger the scale of the
break. We simply cannot go around making wholesale breaks on an ongoing basis.
If we did so then OpenSSL would be a lot less useful to our users.

This is not to say that we can *never* make breaking changes. Only that when we
do so it must be strongly justified and only done relatively infrequently. We
made such a decision when we decided to make the structures opaque. It's not a
decision we are likely to repeat anytime soon IMO. We are still feeling the pain
of that now (and will continue to do so for at least the next year until 1.0.2
goes out of support - and probably beyond that).

Which brings me onto why structures were made opaque in the first place. A
significant driver for this (probably *the* most important one) was to improve
the codebase. I have witnessed first hand the harm that non-opaque structures
did to OpenSSL. We will be fixing the fallout from them for years to come.
Non-opaque structures combined with the requirements for stable API/ABI means
you cannot change anything in those structures. Renaming or deleting structure
members constitutes an API break. Even *adding* structure members constitutes an
ABI break (due to the changed size of the structure). This means the code
ossifies over time and cannot easily be refactored. Much of OpenSSL's internal
"quirkiness" results from attempting to work around this restriction.

Things like the state machine refactor and the record layer refactor would not
have been possible without opaque structures. In my mind making the structures
opaque was one of the best things that ever happened to OpenSSL.


> 
>> https://breakpoint.cc/openssl-1.1-rebuild-2016-08-26/
> 
> So since when do we need to consider API/ABI compatibility? Did we grow up
> recently?
> 
> Or maybe OpenSSL should have switched the language. The point of C is that
> structures are public. And if I'm going to be honest that approach saved my
> sorry arse more than a couple times. When zlib choked because it couldn't go
> past 4 GiBs of data since its fields were uint32_ts, I was able to easily
> workaround this problem. But what do I know.

If you really want to fiddle with OpenSSL internal structures - feel free. Just
include the OpenSSL internal header files and away you go. Just do so in the
knowledge that they could be changed at any time, and your code might break. If
this isn't a concern to you then - no problem. If it is a concern to you - then
actually you *do* care about API/ABI stability after all.


> 
>> Counting symbols is, however, nothing other than a blunt instrument.
>> Quite a lot of those symbols are convenience macros and functions that
>> have accumulated over time.
> 
> You're taking my statement out of context. Counting the symbols wasn't supposed
> to suggest that there are too *many* of them. I'm in no position to say that,
> seeing as the original context in which my statement was put is that *I'm not
> familiar enough with the library*.
> 
> What I said was that reading the code is easy. Learning what the library
> provides is hard, and that you won't learn much just by looking at the symbols
> because there's so many of them.
> 
>> But nevertheless, I do hear you call for a remake of the SSL API as
>> well as cleaner internals.  The latter is easier, and I'm sure it will
>> happen piecemeal as per usual so as to not break something /
>> inadvertently change a behavior (i.e. break ABI).  The former is a
>> fairly massive project, and is more of creating a new API and library
>> rather than a mere cleanup job.  That will be a massive effort, and
>> you do have to keep in mind how much time all involved can put into
>> it.
> 
> I'm not saying it needs to be done right now. I'm merely suggesting that it
> might be a good goal post for OpenSSL 2.0.
> 
>>> Turning structures opaque doesn't prevent people from still messing
>>> with their internal fields.
>>
>> True.  But it makes for a clear delineation where people are forced to
>> be aware that they are playing with internal stuff, and that it may
>> not be a safe thing to do.
> 
> Then why not provide small helper functions for covering the "playing with
> internal stuff" part? That way it's still controlled, and documented, and
> unified. You guys must've had some examples to show off in order to justify the
> process, so surely you know what it is that people do when they use internal
> stuff. Make functions for those. Don't give them any reason to continue playing
> with internal stuff.
> 
> I don't like code that tries to protect programmers from themselves. I like code
> that lets good programmers do smart things. And if bad programmers use that
> freedom to do bad stuff, then doesn't that mean your API simply didn't support
> this, and they had to make it work somehow else? Again, helper functions.
> 
>> Uhmmmm....  this is factually incorrect.  OpenSSL doesn't use its own
>> memory pooling.  We have thin wrappers around the usual malloc() /
>> realloc() / free(), which allows any application to do its own memory
>> pooling.
> 
>>
> https://web.archive.org/web/20150207180717/http://article.gmane.org/gmane.os.openbsd.misc/211963

The BUF_FREELISTS code that this post references was ripped out years ago. This
no longer represents the current state of OpenSSL in any supported version.

>> To conclude, I have a question for you: are you only willing to rant
>> (*), or are you willing to help out in another way?
> 
> This is not the question I feel you should ask because we haven't even
> established if I *could* make contributions to the project, as my mindset
> appears to be so much more different. Especially the idea of not wanting to
> break APIs/ABIs is a huge limitation - just looking at SSL_new() made me give up
> hope here.

Yes - not breaking APIs/ABIs is a huge limitation. BoringSSL is not suitable for
general purpose use precisely because of this. The only users BoringSSL cares
about whether they break or not are Google users. As soon as you have a library
that wants to cater for large numbers of users (which we do) then you have to
accept that limitation.

As to whether or not we have established whether you *could* make such
contributions - I think you are missing the point. We cannot know whether you
are capable or not until you try. It is on the basis of your code that we would
make such a judgement. In order for your code to get into OpenSSL it must have
been reviewed and approved by two current OpenSSL committers (one of whom must
be on the OpenSSL Management Committee). We invite anyone to contribute. In
order for this to be a healthy open-source community we *need* those
contributions. Only those that make the grade will make it in.

Note - this review process wasn't always the case. Things used to be much more
informal in pre-heartbleed days. This is no longer the case.

> 
> I'm no cryptography expert, I've made that clear from mail one, and my cleanup
> jobs would be more widespread than what seems to be deemed acceptable right now.
> I can read and write scalable C code, otherwise I wouldn't even have tried to
> reuse that SSL object from the beginning.
> 
> So, I ask a question in return: what do you think I *could* be helping with?

Well, you have vocally complained about the state of the documentation. You have
the benefit of being a new OpenSSL user. You know what things were confusing or
unclear in the documentation. More experienced OpenSSL coders often don't have
the perspective - because some things are just "obvious" to them. So help with
pull requests to improve the documentation.

Matt