[openssl-users] Authentication over ECDHE

[Richard Levitte]

I'll go ahead and ask, how long do you think such a back door would
stay unnoticed, let alone survive?  I'm considering the fact that we
have a lot of people looking at our code, just judging from the issues
and pull requests raised on github.

I can't say that I have an actual answer, but it's a question worth
asking as well, to see if the T.O.L.A. Act is worth a state of panic
or not.

Cheers,
Richard ( who's on vacation and should stop reading these mails )

In message <11EDC036-87D3-4757-AD33-80C6E608F77F at foocrypt.net> on Tue, 1 Jan 2019 01:27:53 +1100, "openssl at foocrypt.net" <openssl at foocrypt.net> said:

> Matt et al
> 
> 'been reviewed and approved by two current OpenSSL committers (one of whom must
> be on the OpenSSL Management Committee).’
> 
> Due to the recent legislative changes here in Australia around the T.O.L.A. Act, can a change be
> made to the OpenSSL policy so that the 2 reviewers, don’t reside in Australia, or are Australian
> citizens ?
> 
> ABI/API changes -> breaks -> back door requests….
> 
> --
> 
> Regards,
> 
> Mark A. Lane
> 
> Cryptopocalypse NOW 01 04 2016
> 
> Volumes 0.0 -> 10.0 Now available through iTunes - iBooks @
> https://itunes.apple.com/au/author/mark-a.-lane/id1100062966?mt=11
> 
> © Mark A. Lane 1980 - 2019, All Rights Reserved.
> © FooCrypt 1980 - 2019, All Rights Reserved.
> © FooCrypt, A Tale of Cynical Cyclical Encryption. 1980 - 2019, All Rights Reserved.
> © Cryptopocalypse 1980 - 2019, All Rights Reserved.
> 
>  On 1 Jan 2019, at 01:11, Matt Caswell <matt at openssl.org> wrote:
> 
>  On 31/12/2018 11:36, C.Wehrmeyer wrote:
> 
>  On 31.12.18 10:12, Richard Levitte wrote:
> 
>  Yes, it's true, new features are going in. And it's true that it's
>  often more exciting to add new features than to do the janitorial
>  work.
> 
>  You realised what I have left unspoken thus far, which is this almost
>  obsession-like preference of OSS coders to add new features rather than
>  improving the old, boring codebase. However, there's a reason why it's still
>  called code*base*. It's the base of everything. And adding more and more
>  features to that base is going to make ripping off the band-aid more painful in
>  the long run.
> 
>  There has been a huge amount of effort put in over the last few years to improve
>  the codebase. Things that immediately spring to mind (and there's probably a
>  whole load more):
> 
>  - Rewrite of the state machine
>  - libssl record layer refactor
>  - Implementation of the PACKET and WPACKET abstractions in libssl
>  - Rewrite of the rand code
>  - Implementation of the new test harness
>  - Significant effort into developing tests
>  - Implementation of the coding style and reformat of the codebase to meet it
>  - Opaque many of the structures (which I know you don't see as an improvement,
>  but I'll answer that point separately)
>  - Implementation of continuous fuzzing
>  - Significant expansion of the documentation coverage
> 
>  It is simply not true to claim that we have "an obsession-like preference...to
>  add new features rather than improving the old, boring codebase". None of the
>  above things resulted in or were motivated by user visible features. They were
>  all about improving the codebase.
> 
>  Also, infrastructure again. I, as a user, don't care if the kernel gets a new
>  feature that makes some black magic happening. What I care about is that the
>  kernel doesn't throw away my writes (which has happened in May of 2018, see):
> 
>  https://www.postgresql.org/message-id/flat/CAMsr%2BYE5Gs9iPqw2mQ6OHt1aC5Qk5EuBFCyG%2BvzHun1EqMxyQg%40mail.gmail.com#CAMsr+YE5Gs9iPqw2mQ6OHt1aC5Qk5EuBFCyG+vzHun1EqMxyQg@mail.gmail.com)
> 
> 
>  Cryptography libs should be equally conservative, considering that cryptography
>  is conservative to begin with. I don't care if TLS 1.3 lets me use new exiting
>  ciphers and handshakes when it unreasonably bogs down my server code.
> 
>  BUT, you also have to appreciate that stuff is happening around us
>  that affects our focus. TLS 1.3 happened, and rather than having to
>  answer the question "why don't you have TLS 1.3 yet?" (there's a LOT
>  of interest in that version), we decided to add it.
> 
>  Sure, but didn't Matt just say that there are a lot of volunteers working on
>  that library? The disadvantage here is that quality assurance is barely a thing
>  - however, the *advantage* of this is that OpenSSL does not have to follow
>  commercial interests. If we look at this at face value you could just say "No,
>  people, it's high time we streamline some of the internal aspects of the
>  library, TLS 1.3 will have to wait. You can't wait that long? Well, sorry".
> 
>  However, your message is clear, we do need to do some cleanup as
>  well. More than that, I agree with you that it's needed (I've
>  screamed out in angst when stumbling upon particularly ugly or
>  misplaced code, so the feeling is shared, more than you might
>  believe).
> 
>  But what does "cleanup" entail? That's the hot-button question here. I've
>  already made a suggestion, that is to say, getting rid of opaque structures. If
>  that is deemed too insecure (for whatever reasons), export symbols that allow
>  programmers to query the size of structures, and provide two versions of
>  functions: one function expects the caller to pass an object to which the
>  changes are to be made, and the other one allocates the necessary memory
>  dynamically and then calls the first version. Or just don't allocate my object
>  memory dynamically anymore.
> 
>  That being said, cleanup happens, and documentation happens, in a
>  piecemeal fashion, 'cause that's what most people have capacity for.
> 
>  So, what you're effectively saying is that I'm the first one who ever asked for
>  SSL object reuse, right? Because if piecemeal work happens on the documentation,
>  and Viktor says that it's possible, then surely no one would have ever answered
>  that question on the mailing list and *not* put it piecemeal-ly in the OpenSSL
>  documentation, right?
> 
>  Now, here's something else that you need to consider: API/ABI
>  compatibility needs to be preserved.
> 
>  No it doesn't. We *know* it doesn't. When OpenSSL 1.1 was released it broke all
>  *sorts* of applications out there, because all sorts of applications used struct
>  fields rather than accessors. wget, mutt, neon, python, you name it, you broke it.
> 
>  API/ABI stability is absolutely required. Every time we make a breaking change
>  it is painful for our users - and more pain is felt the bigger the scale of the
>  break. We simply cannot go around making wholesale breaks on an ongoing basis.
>  If we did so then OpenSSL would be a lot less useful to our users.
> 
>  This is not to say that we can *never* make breaking changes. Only that when we
>  do so it must be strongly justified and only done relatively infrequently. We
>  made such a decision when we decided to make the structures opaque. It's not a
>  decision we are likely to repeat anytime soon IMO. We are still feeling the pain
>  of that now (and will continue to do so for at least the next year until 1.0.2
>  goes out of support - and probably beyond that).
> 
>  Which brings me onto why structures were made opaque in the first place. A
>  significant driver for this (probably *the* most important one) was to improve
>  the codebase. I have witnessed first hand the harm that non-opaque structures
>  did to OpenSSL. We will be fixing the fallout from them for years to come.
>  Non-opaque structures combined with the requirements for stable API/ABI means
>  you cannot change anything in those structures. Renaming or deleting structure
>  members constitutes an API break. Even *adding* structure members constitutes an
>  ABI break (due to the changed size of the structure). This means the code
>  ossifies over time and cannot easily be refactored. Much of OpenSSL's internal
>  "quirkiness" results from attempting to work around this restriction.
> 
>  Things like the state machine refactor and the record layer refactor would not
>  have been possible without opaque structures. In my mind making the structures
>  opaque was one of the best things that ever happened to OpenSSL.
> 
>  https://breakpoint.cc/openssl-1.1-rebuild-2016-08-26/
> 
>  So since when do we need to consider API/ABI compatibility? Did we grow up
>  recently?
> 
>  Or maybe OpenSSL should have switched the language. The point of C is that
>  structures are public. And if I'm going to be honest that approach saved my
>  sorry arse more than a couple times. When zlib choked because it couldn't go
>  past 4 GiBs of data since its fields were uint32_ts, I was able to easily
>  workaround this problem. But what do I know.
> 
>  If you really want to fiddle with OpenSSL internal structures - feel free. Just
>  include the OpenSSL internal header files and away you go. Just do so in the
>  knowledge that they could be changed at any time, and your code might break. If
>  this isn't a concern to you then - no problem. If it is a concern to you - then
>  actually you *do* care about API/ABI stability after all.
> 
>  Counting symbols is, however, nothing other than a blunt instrument.
>  Quite a lot of those symbols are convenience macros and functions that
>  have accumulated over time.
> 
>  You're taking my statement out of context. Counting the symbols wasn't supposed
>  to suggest that there are too *many* of them. I'm in no position to say that,
>  seeing as the original context in which my statement was put is that *I'm not
>  familiar enough with the library*.
> 
>  What I said was that reading the code is easy. Learning what the library
>  provides is hard, and that you won't learn much just by looking at the symbols
>  because there's so many of them.
> 
>  But nevertheless, I do hear you call for a remake of the SSL API as
>  well as cleaner internals. The latter is easier, and I'm sure it will
>  happen piecemeal as per usual so as to not break something /
>  inadvertently change a behavior (i.e. break ABI). The former is a
>  fairly massive project, and is more of creating a new API and library
>  rather than a mere cleanup job. That will be a massive effort, and
>  you do have to keep in mind how much time all involved can put into
>  it.
> 
>  I'm not saying it needs to be done right now. I'm merely suggesting that it
>  might be a good goal post for OpenSSL 2.0.
> 
>  Turning structures opaque doesn't prevent people from still messing
>  with their internal fields.
> 
>  True. But it makes for a clear delineation where people are forced to
>  be aware that they are playing with internal stuff, and that it may
>  not be a safe thing to do.
> 
>  Then why not provide small helper functions for covering the "playing with
>  internal stuff" part? That way it's still controlled, and documented, and
>  unified. You guys must've had some examples to show off in order to justify the
>  process, so surely you know what it is that people do when they use internal
>  stuff. Make functions for those. Don't give them any reason to continue playing
>  with internal stuff.
> 
>  I don't like code that tries to protect programmers from themselves. I like code
>  that lets good programmers do smart things. And if bad programmers use that
>  freedom to do bad stuff, then doesn't that mean your API simply didn't support
>  this, and they had to make it work somehow else? Again, helper functions.
> 
>  Uhmmmm.... this is factually incorrect. OpenSSL doesn't use its own
>  memory pooling. We have thin wrappers around the usual malloc() /
>  realloc() / free(), which allows any application to do its own memory
>  pooling.
> 
>  https://web.archive.org/web/20150207180717/http://article.gmane.org/gmane.os.openbsd.misc/211963
> 
> 
>  The BUF_FREELISTS code that this post references was ripped out years ago. This
>  no longer represents the current state of OpenSSL in any supported version.
> 
>  To conclude, I have a question for you: are you only willing to rant
>  (*), or are you willing to help out in another way?
> 
>  This is not the question I feel you should ask because we haven't even
>  established if I *could* make contributions to the project, as my mindset
>  appears to be so much more different. Especially the idea of not wanting to
>  break APIs/ABIs is a huge limitation - just looking at SSL_new() made me give up
>  hope here.
> 
>  Yes - not breaking APIs/ABIs is a huge limitation. BoringSSL is not suitable for
>  general purpose use precisely because of this. The only users BoringSSL cares
>  about whether they break or not are Google users. As soon as you have a library
>  that wants to cater for large numbers of users (which we do) then you have to
>  accept that limitation.
> 
>  As to whether or not we have established whether you *could* make such
>  contributions - I think you are missing the point. We cannot know whether you
>  are capable or not until you try. It is on the basis of your code that we would
>  make such a judgement. In order for your code to get into OpenSSL it must have
>  been reviewed and approved by two current OpenSSL committers (one of whom must
>  be on the OpenSSL Management Committee). We invite anyone to contribute. In
>  order for this to be a healthy open-source community we *need* those
>  contributions. Only those that make the grade will make it in.
> 
>  Note - this review process wasn't always the case. Things used to be much more
>  informal in pre-heartbleed days. This is no longer the case.
> 
>  I'm no cryptography expert, I've made that clear from mail one, and my cleanup
>  jobs would be more widespread than what seems to be deemed acceptable right now.
>  I can read and write scalable C code, otherwise I wouldn't even have tried to
>  reuse that SSL object from the beginning.
> 
>  So, I ask a question in return: what do you think I *could* be helping with?
> 
>  Well, you have vocally complained about the state of the documentation. You have
>  the benefit of being a new OpenSSL user. You know what things were confusing or
>  unclear in the documentation. More experienced OpenSSL coders often don't have
>  the perspective - because some things are just "obvious" to them. So help with
>  pull requests to improve the documentation.
> 
>  Matt
>  --
>  openssl-users mailing list
>  To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
>