Monthly Archives: August 2011

Science or Snake Oil: Empirical software engineering

In my professional lifetime, I’ve seen the emergence, development and adoption into the mainstream of object-oriented development for software. Instead of functions or procedures being the principal unit of abstraction, objects that encapsulate data and operations are now seen as the ‘natural’ abstraction in programs, at least outside the embedded systems domain.   I would be surprised if any university computer science course didn’t teach OO programming and my guess is that students are rarely taught to think about whether or not objects really are the right abstraction.

Now, I think the idea of abstract data types, where data and its related functionality are encapsulated is a pretty natural one. It makes sense when thinking about how to organize some data, to associate the operations with that data. But I have never found thinking about a problem to be solved in terms of objects to be a natural approach. I think about what to do to solve the problem and, this seems to me to be a pretty general trait – we articulate solutions to problems in terms of actions.

The fact that objects are not a natural way of thinking does not mean that they are not a good way of programming. This is not really the point of this post – the point is that OO development has become almost universally accepted  in spite of the fact that virtually all empirical studies on its effectiveness show that OO development has no real advantages over a functional approach.

This paper, which studies development productivity in C and C++ makes the telling point:

We investigated whether C++ improved the productivity of software development. We did not find any empirical evidence of improved productivity.  This is in line with previous studies that to date have found no convincing empirical support in favor of object-oriented development.

The last sentence is the telling one – essentially, there is no science to back up the adoption of OO methods and techniques.

Of course, this does not mean that object-oriented approaches are ineffective – they are obviously useful and usable for large system development. But, we have to ask ourselves if this effectiveness is inherent in the approach or because there has been a vast amount of effort and money invested in the development of tools and methods to support the object-oriented approach. I wonder if we had invested the same amount of money and effort in developing tools and methods for languages such as Modula 2  instead of Java  would we have a better programming environment?

The point of this post is not really about OO development, however. What I find concerning is that empirical  work in software engineering is essentially ignored by almost everyone. The paper I’ve linked above was published in 2008 but Les Hatton wrote a paper  in 1996 about studies of OO development and there may well have been others before that.

One problem is our academic publishing system where interesting results are published in obscure conferences and journals that no-one reads and authors get no academic credit for making their material more accessible to a wide audience. Another problem is our research funding system that focuses on novelty rather than science – it’s very hard to get funding for a project that aims to confirm or deny previous work.

But the biggest problem, I think, are the snake oil salesmen. The sellers of universal solutions who don’t admit there is no scientific basis for the claims they are making, who don’t admit there are some areas where their techniques are inapplicable and who are selective with the truth when discussing the use of their techniques.

We are now seeing another ‘revolution’ in software engineering as the agile methods bandwagon becomes unstoppable. I have a lot of time for some aspects of agile methods but we should be honest about their effectiveness and applicability.

For example, few discussions of test-driven development, a central agile development technique, admit that empirical studies of TDD (in industry (Microsoft and IBM) show that it slows down development – so you might think it is less rather than more agile. These studies also show that it leads to fewer post-release problems, confirming the technique is of value but not, perhaps, for the reasons given by the agile salesmen.

Sadly, there are too many vested interests here – academics seeking tenure and promotion and consultants seeking contracts – that a change to a more scientific approach in the near future is unlikely. The people who will have to pay the price will be the adopters of these techniques who will learn that the magic snake oil is not a solution to their problems after all.


Filed under Uncategorized