Is it possible to validate LSCITS research?

For the past 5 years or so, I’ve been working on a UK research programme of research and education into large-scale complex IT systems (LSCITS). This has involved partners in other universities and industry. Overall, I think we’ve done a good job with lots of interesting research results. Thanks to the flexibility of EPSRC funding, we’ve been able to be responsive to new development that weren’t anticipated when we put the proposal together such as social networking and cloud computing.

You can see a list of what we’ve produced at the LSCITS web site.

So, academically all is well. Lots of publications, students have received PhDs and staff have been promoted. We’ve ran successful workshops and achieved our aim of creating an LSCITS community.

Yet, in spite of this, I am left with a feeling of unease. So far, very few of our results have had any impact on practice. This is not, in itself, a problem as it takes a while after a project finishes before the results can have an impact. But, if and when they are used, how will we know how good they are? I feel uneasy because, frankly, even with commitment and support from industrial users, I have no idea how we can assess the value of our work for improving real large-scale systems engineering practice.

Let us assume that some company or collaboration decides to take some of our ideas on board – let’s say those on socio-technical analysis.  They apply these on a project and eventually go on to create a system that the stakeholders are happy with. Does this mean our ideas have helped? Or, if the project is deemed to be a failure, does this mean that our ideas don’t work?

The problem with large-scale systems is just that – they are large-scale and their size means that there are lots of factors that can affect the success or otherwise of development projects. These factors are present in all projects but the influence of particular factors varies significantly – for example, real-time response is a key success factor in some systems but less important in others. Not only do we not know in advance which factors are likely to be significant, but we don’t really maintain enough information from previous projects even to hazard a guess.  We don’t understand how these factors relate to each other so we don’t know the consequences of changing one or more of them.

So, is it impossible to validate if LSCITS research makes a difference? If so, what is the purpose of doing that research? My answer to the first question is that I think it is practically if not theoretically impossible; the second, I’ll make the topic of another blog post.


Filed under LSCITS, research

5 responses to “Is it possible to validate LSCITS research?

  1. Sounds like a great example of when to use the case study method to do an evaluation.

  2. David

    Lorin, I disagree with your claim that ‘case study method’ would be appropriate to do this. Perhaps I have misunderstood your claim?

    Case studies tend to be for exploratory work (discovering hypotheses to test) rather than for validating hypotheses. I’d have thought some sort of factor analysis over a large population of projects would be more helpful in validating whether specific practices/techniques are associated with certain outcomes.

    See for example the work of Eric Honour in understanding the impact of systems engineering practices on quality and schedule.


  3. David:

    Case studies have been used for evaluation as well as exploratory research.

    The authoritative text on case study research is “Case Study Research: Design and Methods” by Yin, which goes into detail on how to marshall evidence and use triangulation to establish causal links:

    The U.S. Government Accountability Office (GAO) has a text that describes how to run case study evaluations:

    More specific to software engineering, among others, there’s lots of stuff, here are a few links.

    * Runeson & Host: Guidelines for Conducting and reporting case study research in software engineering in Empirical Software Engineering
    * Kitchenham, Pickard and Pfleeger “Case Studies for Method and Tool Evaluation” from IEEE Software, July 1995
    * Kitchenham “DESMET: a methodology for evaluating software engineering methods & tools” which discusses using (quantitaitve) case studies in addition to experiments.
    * Shneiderman and Plaisant have advocated for long-term case study evaluations for evaluating info viz software:

  4. r. schaefer

    Do these references address handling the politics of large organizations? Any case study of a large project would have to get involved with the politics of the organizations that host the software development. “Bad news” that points to organizations and management would most likely get suppressed.

    • I think that ‘bad news’ gets suppressed in systems in private industry but less so in government system inquiries. Completely agree that politics are often the major barrier to success

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s