Abstraction and complexity

I gave a talk recently about complex systems engineering at Stirling University where I discussed my notions that software engineering is essentially reductionist and we need to rethink software engineering approaches to cope with the complex systems that we are now building. I was challenged by a questioner who claimed that abstraction was an effective way to deal with complexity and I’m afraid that I dismissed this rather glibly without any real rationale of why it was inappropriate.

I have now thought about this and I now think that I can present a better rationale of why abstraction is ineffective for complexity management. In a nutshell, complexity arises because of the interactions between the elements of a system (see my blog post on complexity). Systems are inherently complex when these interactions are dynamic and where they change their nature over time and in response to environmental stimuli. Complicated systems are ones where there are many elements, perhaps of different types and where elements may have many distinct characteristics but where the relationships between these elements are static.  For example, a topographic map is complicated but it is not complex.

Abstraction, however, is a mechanism for dealing with diversity in the system elements where abstractions represent the essential (for that system) characteristics of a collection of elements. Therefore, if we are building a transport model, we may have an abstraction ‘car’ which has characteristics of size and speed – we don’t care about marque, colour, etc. This is an absolutely essential mechanism for understanding and reasoning about systems and for helping us create software – but it helps us deal with complicated not complex systems.



Filed under LSCITS

8 responses to “Abstraction and complexity

  1. Dave Greenwood

    I agree with you Ian but also add that the essential tools to understand complexity as you define it “complexity arises because of the interactions between the elements of a system” already exists in other disciplines.

    As you may be aware, in operations research “complexity [that] arises because of the interactions between the elements of a system” is called system dynamics. The ‘system dynamics’ tradition of analysis has proved useful to analyse a myriad of complex systems including socio-technical systems. Wikipedia link here: http://en.wikipedia.org/wiki/System_dynamics

    What needs to happen is for techies to translate System Dynamics into CS concepts and then use it to complement (UML based or what ever) structural and behavioural modeling. The uptake of System Dynamics, I think, will be inevitable since IT is becoming exhibiting increasing behavioural dynamics e.g. growth of SoS and Cloud.

  2. I’m afraid I’m siding with your questioner! Abstraction is practically the only method we have of partitioning large and complex systems. In Executable UML, not only are classes identified but also relationships and more importantly, for the purposes of this discussion, their behaviour in the form of state machines – which can be used to manage dynamic relationships.

    Your notion of complicated systems seems to equate to how much data the system is handling. If this is a correct interpretation I don’t think it’s relevant to a discussion of complexity.

  3. Dave Greenwood

    Hi Mike,

    I think you may have misunderstood what is meant by ‘complexity’. What we mean by ‘complexity’ (in LSCITS) is that the behaviour of a system is nonlinear I.e. The behaviour of the system is more than the linear sum of its parts. For a more technical definition of a nonlinear systems see: http://en.wikipedia.org/wiki/Nonlinear_system

    I’m not sure that a state machine model is helpful for modelling nonlinear systems without first understanding the nature of its interactions and the possible cascades or ripple effects that may occur. Once you’ve understood the interactions you may well decide to model the behaviour of the system using a (probablistic) state machine. And perhaps use a probablistic model checker, like PRISM, to assert some properties of the system.

    Ian is suggesting, I think, that current SE abstractions pay insufficient attention to the nature of (nonlinear) interactions and are thus unsuitable for modelling systems that exhibit non-linear behaviour.

    I tentatively suggest that by translating ‘system dynamics’ into a form that is compatible with UML it will enable software engineers to get a better understanding of nonlinear systems behaviour. Another interesting take is the performance analysis view found in SysML (a modified version of UML for systems engineering).

    • Hi Dave,

      Apologies for my tardy response but I’ve been out of action for the last few days.

      Thanks for your informative reply and you’re correct in assuming that I was unaware of the LSCITS definition of complex. I did not recognise the significance of the “Filed under: LSCITS” link but have now had time to read up on the subject and to mull the problem over.

      I must admit to having a problem with the LSCITS definition of complex. The definition seems to rely on a computer program being somehow inherently non-deterministic which should not be the case. I am using the term “computer program” to mean all software running on hardware.

      Excluding the possibility of error conditions such as memory corruption caused by rouge atomic particles or gamma rays, a computer program and its interaction between other computer programs should be absolutely stable (not show any chaotic behaviour). It’s the job of the Analyst to find and eradicate such behaviour without mercy.

      It seems that LSCITS discusses two classes of computer systems. The first is the black-box type where the various communicating systems are largely an unknown quantity and we must observe how they react to various stimuli to figure out their behaviour. I suspect access to the source code will not help much with these systems since it would difficult to grasp what is going on inside due to the lack of separation of concerns and the sheer amount of code, data and processes involved.

      I believe LSCITS describes this type of system as non-linear because of the difficultly of understanding and also because it might exhibit genuinely chaotic behaviour of a non-linear system. I think this is the root of the mysterious emergent behaviour that appears (only) to the black box observer.

      The second is the white-box type where some analysis formalism, possibly based on the Relational Theory of Data, State Machines and Petri Nets has been used by an Analyst to design the various communicating systems that make the combined system. Using a formalism (modelling) the Analyst can have a far better understanding of what’s going on in the system.

      However, I suspect white-box systems are few and far between. The purpose here for the formalism is to enable the Analyst to exclude any undesirable behaviour by synchronising systems (at the state machine level). I’m not saying that the Analyst will get this right first time because synchronisation is probably the most difficult problem for developers to deal with even with the aid of a formalism such as Executable UML. But by making use of simulation within an appropriate environment the Analyst should understand and eliminate any synchronisation problems, eventually.

      The idea of the analysis formalism is to turn a complex system into a complicated one.

  4. If instead of the ‘complex systems’ defn. of complexity, I take the common English distinction between complicated (as in made complex) and compelx (as in intrinsically so) – rather like simplistic vs. simple; then the purpose of abstraction is clear – to avoid complicated things. The description of the world should be no more complicated than it needs to be.

    However, even with intrinsically complex things – this is not new- this world we live in is complex and yet our ability to abstract has developed precisely to deal with this complexity.

    So what is different now?

    Perhaps first there are more complicated things that we believe to be complex – the agglomeration of off the shelf components are a case in point – these are only complex (in the technical complex systems sense) because they are complicated (in the common English sense). This seems exactly where abstractions should be useful.

    However, there are things that are intrinsically complex – and always have been – the weather, the environment, the shape of the land beneath our feet … other people.

    The difference today is perhaps that we seek to predict, manage and control in a way our forbearers never expected to. Even then, we do use abstractions, for example, naming different shapes of clouds, and build predictive (but not deterministic) rules based on them.

    And when these rues failed the solution has always been simple – engineering – changing the word to fit abstractions, cultural practices and hierarchical organisations for managing people, roads on the ground to make the wheel work better.

    The philosophical question is “what abstractions fit the world”

    The practical question is “how do we make the world fit useful abstractions”

    While there certainly are places where we reach our limits, many of those where we see “complexity” are really just plain complicated.

  5. Alan – I agree with you. But if something is so complicated that you can’t practically understand or model it, is there really a difference. See my thoughts on this:

  6. Our society is demanding a (complex) networked system with humans and machines interacting all the time… Certainly, individual elements of this network can be designed and controlled by well-known tools (like abstraction). I think the whole network is too much complex to be modelled consistently in 2D or 3D views ammenable for most people. Why? Because the global architecture doesn’t exist or doesn’t stabilize if its constituent subarchitectures and components are growning and adapting quickly. Governments and businesses can not control or preview the next month’s behaviour of this kind of system because people are changing data and data are changing people which changes the (software/hardware) applications available in that networded system.
    Evolutionary design, an agile discipline, is a strong tool to extend the software life-cycle.

  7. IMHO there are several false distinctions being made here. Every model is an abstraction. Different models from different viewpoints, modelling different aspects of the system, are different abstractions. How else can we hope to understand the aspects of a complex system that matter to us – whether they are existential questions (is that thing over there interested in eating me? Could it catch me if I ran?, what would happen if I threw a rock at it?) or artistic ones (why does that music sound mournful?), or scientific ones (how many stable isotopes of hydrogen are possible?).

    If anyone knows another way of working with complexity, please say what it is and justify why it does not involve abstraction.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s