Science or Snake Oil: Empirical software engineering

In my professional lifetime, I’ve seen the emergence, development and adoption into the mainstream of object-oriented development for software. Instead of functions or procedures being the principal unit of abstraction, objects that encapsulate data and operations are now seen as the ‘natural’ abstraction in programs, at least outside the embedded systems domain.   I would be surprised if any university computer science course didn’t teach OO programming and my guess is that students are rarely taught to think about whether or not objects really are the right abstraction.

Now, I think the idea of abstract data types, where data and its related functionality are encapsulated is a pretty natural one. It makes sense when thinking about how to organize some data, to associate the operations with that data. But I have never found thinking about a problem to be solved in terms of objects to be a natural approach. I think about what to do to solve the problem and, this seems to me to be a pretty general trait – we articulate solutions to problems in terms of actions.

The fact that objects are not a natural way of thinking does not mean that they are not a good way of programming. This is not really the point of this post – the point is that OO development has become almost universally accepted  in spite of the fact that virtually all empirical studies on its effectiveness show that OO development has no real advantages over a functional approach.

This paper, which studies development productivity in C and C++ makes the telling point:

We investigated whether C++ improved the productivity of software development. We did not find any empirical evidence of improved productivity.  This is in line with previous studies that to date have found no convincing empirical support in favor of object-oriented development.

The last sentence is the telling one – essentially, there is no science to back up the adoption of OO methods and techniques.

Of course, this does not mean that object-oriented approaches are ineffective – they are obviously useful and usable for large system development. But, we have to ask ourselves if this effectiveness is inherent in the approach or because there has been a vast amount of effort and money invested in the development of tools and methods to support the object-oriented approach. I wonder if we had invested the same amount of money and effort in developing tools and methods for languages such as Modula 2  instead of Java  would we have a better programming environment?

The point of this post is not really about OO development, however. What I find concerning is that empirical  work in software engineering is essentially ignored by almost everyone. The paper I’ve linked above was published in 2008 but Les Hatton wrote a paper  in 1996 about studies of OO development and there may well have been others before that.

One problem is our academic publishing system where interesting results are published in obscure conferences and journals that no-one reads and authors get no academic credit for making their material more accessible to a wide audience. Another problem is our research funding system that focuses on novelty rather than science – it’s very hard to get funding for a project that aims to confirm or deny previous work.

But the biggest problem, I think, are the snake oil salesmen. The sellers of universal solutions who don’t admit there is no scientific basis for the claims they are making, who don’t admit there are some areas where their techniques are inapplicable and who are selective with the truth when discussing the use of their techniques.

We are now seeing another ‘revolution’ in software engineering as the agile methods bandwagon becomes unstoppable. I have a lot of time for some aspects of agile methods but we should be honest about their effectiveness and applicability.

For example, few discussions of test-driven development, a central agile development technique, admit that empirical studies of TDD (in industry (Microsoft and IBM) show that it slows down development – so you might think it is less rather than more agile. These studies also show that it leads to fewer post-release problems, confirming the technique is of value but not, perhaps, for the reasons given by the agile salesmen.

Sadly, there are too many vested interests here – academics seeking tenure and promotion and consultants seeking contracts – that a change to a more scientific approach in the near future is unlikely. The people who will have to pay the price will be the adopters of these techniques who will learn that the magic snake oil is not a solution to their problems after all.



Filed under Uncategorized

10 responses to “Science or Snake Oil: Empirical software engineering

  1. Thomas Owens

    ” would be surprised if any university computer science course didn’t teach OO programming and my guess is that students are rarely taught to think about whether or not objects really are the right abstraction.
    ” See – Carnegie Mellon University no longer teaches OO programming in their introductory curriculum.

    • I saw this. But their announcement also said:

      “We certainly haven’t stopped teaching or using Java, which was developed by one of our alumni, James Gosling, and is widely used by software developers,” Bryant said. “But we believe other programming languages do a better job of helping beginning students understand the principles underlying this science”

  2. Very good point on Agile. I’m a consultant (or vested interest) and I try to fairly present TDD and other things to clients with the caveats you listed. There is no silver bullet!
    But many times my clients are so thirsty for any answer to their problems and having heard of agile before, that they drown out all caveats I present. Frustrating, but we still seem to be in the “Just make it work” phase and have not progressed to the “How can we make it better” phase.

  3. Dave Greenwood

    The fundamental problem that Ian highlights is a prime example of the ‘garbage can’ model of organisational choice. In many organisations there are:
    1) people with ‘problems’;
    2) people with ‘solutions’ that need ‘problems’ to solve;
    3) people with issues and views to express.

    The end result is that decisions are not made on technical merit but political expedience.

    See Cohen, March, Olsen (1972)
    Weick (1995) Sensemaking in Organisations

  4. OO is a tool and not he way. The master guru chooses the tool that most directly addresses the problem.

    At Bell Labs, for those partitions of a solution suited to OO, I measured a 3:1 productivity improvement in the reuse of object libraries and no improvement over logical programming for first use of a customize solution. I concluded OO was best used in product line development and for the admin and provisioning partitions of a software platform.

    Intricate OO can make multicore programming impossible.

    Larry Bernstein

  5. Nelly BEncomo

    Pete (Sawyer) says you are totally consistent with what you have said 20 years ago. I think you may also be right!

  6. Hi Ian!

    I liked the article, maybe because I agree with what you say 🙂 It seems to me however that there is a recent trend to encourage empirical work in software engineering. If this trend continues we’ll have less snake oil in the future and the discipline can start being more reason-based.


    • I agree we are seeing more empirical work. Journals are the best outlet for this, not short conference papers (see my post on this topic). With journal publications, people who make measurements can go into more depth about what these measurements mean and, critically, they can make their raw data available to others. If others can repeat the analysis, this increases confidence in the results.

  7. Pingback: TDD: Links, News and Resources (2) « Angel “Java” Lopez on Blog

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s