Category Archives: Uncategorized

Getting back to the sharp end

The nature of careers is often that, as you get older, you spend more time managing, negotiating and facilitating rather than actually doing software engineering. I’m no exception to this and I haven’t had operational involvement in a significant systems project for quite some time. However, I’ve now had the opportunity of getting involved again in some real systems requirements engineering.

I’ve been asked to lead a group looking at the architectural requirements for a replacement system for Glow – a system that’s supposed to support collaborative learning and resource sharing in all Scottish schools. The Glow system does not have a great reputation amongst teachers and we hope that the replacement system will be more acceptable and better suited to what they need.

I’m at the stage at the beginning of the project where I’m overwhelmed by the complexity of the problem and it’ll take a bit of time to work through this and get started.  As always, the real problems with this system are not technical but are political  – senior politicians will have egg on their face if this goes wrong and the governance of the system is split between a number of different bodies. There’s a very wide variety of stakeholders and many, maybe most users, frankly, couldn’t care less if the system is replaced or not.

Luckily, this has come up at a time when I’ve just finished a major job and hadn’t yet started anything else so I will be able to give it some time. We have a group of motivated and enthusiastic teachers involved and at least one very articulate student user. This is a major challenge but I’m very excited by the prospect of getting back to the sharp end and doing some practical engineering.

So, when it comes to the next edition of Software Engineering,  expect a new case study on IT systems for education.

Advertisements

2 Comments

Filed under Uncategorized

The IEEE simply doesn’t get open access publishing

I think that open-access publishing where research papers are freely accessible on the web is ultimately inevitably although it will take a while to break the monopoly of the big academic publishers (see this article by George Monbiot to see how bad this is).  Ethically, if researchers are paid to do research through public funds, then the public have a right to access their work.

Organising publication and peer reviews is not free, however, so a fee has to be paid by the author. This is a good thing in my view as it means that authors will think twice before submitting work that is not mature enough and may ultimately reduce the total number of papers published.

Now most journals as run by volunteers so, typically the editor and reviewers are not paid. So the principal cost is perhaps some administrative support for the editor.  If we assume that the cost of a full-time administrator + office space/phone/computer is about £60, 000 per year and a journal publishes 10 papers per month for 12 months, then the publication cost should be around £500. Details don’t matter – the essence is that the costs are hundreds not thousands of pounds/dollars/euros. 

The IEEE recently announced that it would provide what they call ‘hybrid open access’ where authors can opt for their papers to be openly available, at a cost of $3000.  However, the other papers in the journal are still only available to subscribers and, of course, there are no signs of the subscriptions being reduced.  If $3000 is the real cost (taking into account that advertising and subscriptions also contribute) then it looks to me like the IEEE is a pretty inefficient organization.

But the point here is that hybrid open access is complete nonsense. Either a journal is open access – available freely to all or it isn’t, There is no halfway house that makes sense – its unfair on authors to charge them as if there were no subscribers and its unfair on subscribers not to reduce their subscription if some articles are open-access. 

We see more evidence that the IEEE doesn’t understand the model (or, more likely, is doing all it can to stop open-access publishing) in its idiotic statement about additional page costs

“Overlength page charges support the costs of printing extra pages above a standard length set by the journal”

Open access is NOT about print publishing – printed journals are an anachronism and I’m willing to bet that the majority of IEEE journal users don’t get printed versions. 

I’ve been a member of the IEEE for almost 30 years and I like their journals. I think that professional societies should be the leaders in promoting open access and I’m saddened by the fact that either they don’t understand this or that they wish to discourage this approach to publishing.

Personally, I’ve recently taken the position that I won’t review for journals that don’t make articles freely available and I won’t submit articles to them. See:

http://www.researchwithoutwalls.org/

2 Comments

Filed under Uncategorized

Original ideas don’t get published in top conferences

I recently read a blog post on why papers were rejected for the ACM conference on Human Computer Interaction (CHI 2012). I found this to be profoundly depressing as it came out with a list of pseudo-scientific reasons why papers were rejected, without taking any account of the fact that demonstrating new ideas is not science.

I’m fairly sure that the author of the post was trying to be helpful and was well-intentioned but when these are the reasons for rejection, then it means it is highly unlikely that original ideas will be published. Imagine if Apple had submitted a paper on the ipod to this conference – no formal evaluation so not wanted!  Rather, the ideal paper is an increment on something that has already been done and which is then analysed to death. Whether it is interesting or exciting doesn’t matter.

I’m not trying to get at the CHI community here – the situation is exactly the same in my area of software engineering. Papers for the ICSE conference are profoundly dull. In fact the only reason to attend this conference (and I guess CHI) is that there are lots of interesting workshops going on alongside them where people actually talk about ideas.

Contrast this situation with journal publication. I recently completed the 3rd revision of a paper that has been accepted for the CACM. I never bothered submitting this to any conference because it contained some controversial ideas and I knew that a reviewer would disagree with them and they were rejected. Not surprisingly, a reviewer of the journal paper also disagreed – but the result was not to recommend rejection but to ask for more details and clarification. Essentially, journal papers are refined through a conversation between the authors, reviewers and editors. And they are all the better for this.

Of course there is a place for papers that are incremental, with lots of analysis. That place is in journals where readers who are interested have time to read and digest the analysis. Conferences are where people get together and they should be exciting and should stimulate lots of discussions. This means their focus should be on ideas and originality, so that people argue about them and generate their own ideas.

But at the moment, if you have a new idea, don’t waste your time submitting it to a top CS conference. These have become vehicles for academics who want promotion or tenure and they have completely lost sight of what conferences should be about.

18 Comments

Filed under Uncategorized

A research agenda for software engineering

I have written elsewhere in this blog on how I believe that reductionism as a basis for software engineering is not appropriate for the complex coalitions of systems that we are now building. This means that ‘more of the same’ – improving existing software engineering methods – simply will not allow us to tackle the development of the large-scale, complex software systems that are rapidly becoming a reality.

Rather, I believe that we need to completely rethink software engineering research and propose the following top-10 research problems that I think need to be addressed:

1.      How can we model and simulate the interactions between independent systems?
To help us understand and manage coalitions of systems we need dynamic models that are updated in real-time with information from the actual system. We need these models to help us make rapid ‘what-if’ assessments of the consequences of system change options. This will require new performance and failure modelling techniques where the models can adapt automatically from system monitoring data.

2.      How can we monitor coalitions of systems and what are the warning signs of problems?

To help avoid the transitions to sunstable system state, we need to know what are the indicators that provide information about the state of the coalition of systems, how these indicators may be used to provide both early warnings of system problems and, if necessary, switch to safe-mode operating conditions that will stop damage occurring. To make effective use of this data, we need visualization techniques that reveal the subtleties of coalition operation and interactions to operators and users.

3.      How can systems be designed to recover from failure?

As we construct coalitions of systems with independently-managed elements and negotiated requirements, it is increasingly impractical to avoid ‘failure’. Indeed, what seems to be a ‘failure’ for some users may not affect some others.  Because some failures are ambiguous, automated systems cannot cope on their own. Human operators have to use information from the system and intervene to recover from the failure and restore the system. This means that we need to understand the socio-technical processes of failure recovery, the support that these operators need and how to design coalition members to be ‘good citizens’ and to support failure recovery.

4.      How can we integrate socio-technical factors into systems and software engineering methods?

Software and systems engineering methods have been created to support the development of technical systems and, by and large, consider human, social and organisational issues to be outside the system boundary. However, these non-technical factors significantly affect the development, integration and operation of coalitions of systems. There is a considerable body of work on socio-technical systems but this has not been ‘industrialised’ and made accessible to practitioners.

5.      To what extent can coalitions of systems be self-managing?

The coalitions of systems that will be created are complex and dynamic and it will be difficult to keep track of system operation and respond in a timely way to the monitoring and health measurement information that is provided. We need research into self-management so that systems can detect changes in both their own operation and in their operational environment and dynamically reconfigure themselves to cope with these changes. The danger is that reconfiguration will create further problems so a key requirement is for these techniques to operate in a safe, predictable and auditable way and to ensure that self-management does not conflict with ‘design for recovery’.

6.      How can we manage complex, dynamically changing system configurations?

Coalitions of systems will be constructed by orchestration and configuration and the desired system configurations will change dynamically in response to load, indicators of the system health, unavailability of components and system health warnings. We need ways of supporting construction by configuration, managing configuration changes and recording changes (including automated changes from the self-management system) in real-time so that we have an audit trail recording what the configuration of the coalition was at any point in time.

7.      How can we support the agile engineering of coalitions of systems?

The business environment changes incredibly quickly in response to economic circumstances, competition and business reorganization. The coalitions of systems that we create will have to change rapidly to reflect new business needs. A model of system change that relies on lengthy processes of requirements analysis and approval simply will not work.

Agile methods of programming have been successful for small to medium sized systems where the dominant activity is system development. For large and complex systems, development processes are often dominated by coordination activities involving multiple stakeholders and engineers who are not co-located. How can we evolve agile approaches that are effective for ‘systems development in the small’ to support multi-organization, global software development?

8.      How should coalitions of systems be regulated and certified?

Many coalitions of systems will be critical systems whose failure could threaten individuals, organizations and economies. They may have to be certified by a regulator who will check that, as far as possible, the systems will not pose a threat to their operators or the wider systems’ environment. But certification is increasingly expensive. For some safety-critical systems the cost of certification may exceed the costs of development. These costs will continue to rise as systems become larger and more complex.

9.      How can we do ‘probabilistic verification’ of systems?

Our current techniques of system testing and more formal analysis are based on the assumption that the system has a definitive specification and that behaviour which deviates from that specification can be recognized. Coalitions of systems will have no such specification nor will system behaviour be guaranteed to be deterministic. The key verification issue will not be ‘is the system correct’ but ‘what is the probability that it satisfies essential properties, such as safety, that take into account its probabilistic, real-time and non-deterministic behaviour’.

10.    How should shared knowledge in a coalition of systems be represented?

We assume that the systems in a coalition will interact through service interfaces so there will not be any over-arching controller in the system. Information will be encoded in a standards-based representation.  The key problem will not therefore be a problem of compatibility – it will be a problem of understanding what the information that systems exchange actually means.

Currently, we address this problem on a system by system basis with negotiations taking place between system owners to clarify what shared information means. However, if we allow for dynamic coalitions with systems entering and leaving the coalition, this is no longer a practical approach.  The key issue is developing a means of sharing the meaning of information – perhaps using ontologies as proposed in the work on the semantic web.

Thanks to Dave Cliff and Radu Calinescu for their input. More on this in our paper on Large Scale Complex IT Systems.

1 Comment

Filed under Uncategorized

Science or Snake Oil: Empirical software engineering

In my professional lifetime, I’ve seen the emergence, development and adoption into the mainstream of object-oriented development for software. Instead of functions or procedures being the principal unit of abstraction, objects that encapsulate data and operations are now seen as the ‘natural’ abstraction in programs, at least outside the embedded systems domain.   I would be surprised if any university computer science course didn’t teach OO programming and my guess is that students are rarely taught to think about whether or not objects really are the right abstraction.

Now, I think the idea of abstract data types, where data and its related functionality are encapsulated is a pretty natural one. It makes sense when thinking about how to organize some data, to associate the operations with that data. But I have never found thinking about a problem to be solved in terms of objects to be a natural approach. I think about what to do to solve the problem and, this seems to me to be a pretty general trait – we articulate solutions to problems in terms of actions.

The fact that objects are not a natural way of thinking does not mean that they are not a good way of programming. This is not really the point of this post – the point is that OO development has become almost universally accepted  in spite of the fact that virtually all empirical studies on its effectiveness show that OO development has no real advantages over a functional approach.

This paper, which studies development productivity in C and C++ makes the telling point:

We investigated whether C++ improved the productivity of software development. We did not find any empirical evidence of improved productivity.  This is in line with previous studies that to date have found no convincing empirical support in favor of object-oriented development.

The last sentence is the telling one – essentially, there is no science to back up the adoption of OO methods and techniques.

Of course, this does not mean that object-oriented approaches are ineffective – they are obviously useful and usable for large system development. But, we have to ask ourselves if this effectiveness is inherent in the approach or because there has been a vast amount of effort and money invested in the development of tools and methods to support the object-oriented approach. I wonder if we had invested the same amount of money and effort in developing tools and methods for languages such as Modula 2  instead of Java  would we have a better programming environment?

The point of this post is not really about OO development, however. What I find concerning is that empirical  work in software engineering is essentially ignored by almost everyone. The paper I’ve linked above was published in 2008 but Les Hatton wrote a paper  in 1996 about studies of OO development and there may well have been others before that.

One problem is our academic publishing system where interesting results are published in obscure conferences and journals that no-one reads and authors get no academic credit for making their material more accessible to a wide audience. Another problem is our research funding system that focuses on novelty rather than science – it’s very hard to get funding for a project that aims to confirm or deny previous work.

But the biggest problem, I think, are the snake oil salesmen. The sellers of universal solutions who don’t admit there is no scientific basis for the claims they are making, who don’t admit there are some areas where their techniques are inapplicable and who are selective with the truth when discussing the use of their techniques.

We are now seeing another ‘revolution’ in software engineering as the agile methods bandwagon becomes unstoppable. I have a lot of time for some aspects of agile methods but we should be honest about their effectiveness and applicability.

For example, few discussions of test-driven development, a central agile development technique, admit that empirical studies of TDD (in industry (Microsoft and IBM) show that it slows down development – so you might think it is less rather than more agile. These studies also show that it leads to fewer post-release problems, confirming the technique is of value but not, perhaps, for the reasons given by the agile salesmen.

Sadly, there are too many vested interests here – academics seeking tenure and promotion and consultants seeking contracts – that a change to a more scientific approach in the near future is unlikely. The people who will have to pay the price will be the adopters of these techniques who will learn that the magic snake oil is not a solution to their problems after all.

10 Comments

Filed under Uncategorized

The future of textbooks. Part 4: Author-led publishing

In my final post on this topic, I put forward my view that the textbook publishing industry, unless it changes radically and quickly, doesn’t have a future. But I also think that we still need textbooks so how will these be ‘published’ and brought to the attention of potential users. In this post, I’ll discuss two possibilities

Open source textbooks

The idea of open-source textbooks is an attractive one.  Like open-source software, the ‘community’ would cooperate to create a textbook that would evolve and develop as people contributed to it. Like open source software, this could not be a free for all but there would have to be some ‘editorial collective’ that decided on which changes to accept and which established the general structure of the book. The Open Text Book repository has a number of open source texts but no indication of how many users there are.  The Connexions project encourages authors to submit modules which can then be assembled into ‘book’, tailored for a particular course.

I’d like to write a textbook on systems engineering, have quite a lot of material but have major gaps in my knowledge and no time to fill these in. So, I have wondered about an open source approach.  But having looked at the topics I understand from existing sites, the quality is not up to my standards and the danger is that with an open-source text I would spend more time moderating contributions than actually learning about and writing the material.

The distinction between open-source software and books is that books are not that large and one individual can produce a book in a reasonable time. I would never say that open-source is an unworkable approach – but I doubt if it will become the normal model for textbook production.

Author-led publishing

Publishers currently support book production, printing, distribution and marketing.  They also supposedly have some quality control functions which some publishers take seriously. Others, such as Springer, in my experience don’t even seem to read the camera-ready texts submitted to them.

Printing and distribution aren’t required for e-books. Software to support production is getting better and better.  Production support such as copyediting is mostly done by freelancers. This leaves marketing which, in my opinion, most publishers don’t do particularly well.

So, why shouldn’t authors do it themselves. They produce the books and take all of the revenue from them – and pay people to do the work.  Currently, authors get well under 10% of the cover price of a textbook – let’s guess £2 for a £40 text. Now sell it at £20 – authors get 10 times their current income. Costs will probably be at least 60% -70% of this – but authors are better off and more importantly are in control.

Of course, there are two arguments against this – firstly, authors are not interested in getting involved – they simply want to hand over their book and let someone else take care of everything. Secondly, it changes the balance of risk – from the publisher to the author. The publisher rather than the author takes the risk that the income from the book will not cover the costs.

The first of these objections is not really a problem. The publishing industry is increasingly a freelance industry anyway and there are lots of people with publishing experience who could set themselves up as micropublishers. We could also see the revival of individual university presses, funded by authors in a university. They key difference between this and the current publishing model is control – authors own their books and pay as required for services.

The second objection is very valid in a world where costs are dominated by printing and distribution. However, with e-books, the absolute costs of production are very small – authors may chose to get professional help with design, copy-editing, etc. but if they expect their sales to be relatively low then they may simply do it themselves. Equally, they may rely on personal contacts and social media for marketing.  For books where sales are likely to be higher, authors may be willing to take more risks and spend more up-front on their book.

Of course, with e-books, a hybrid approach is possible. Low initial investment but if sales generate sufficient revenue, then re-investment in the design, editing and marketing is possible.

In conclusion, the future of textbooks is likely to be one where there is much greater diversity than there is at the moment. I am convinced that e-books will dominate with print-on-demand if paper copies are required. I think it likely that some major publishers will continue to publish texts that sell in relatively large numbers. However, the future for publishers who currently charge high prices for specialized texts, sold in relatively small quantities looks bleak. And, for sure, current high textbook prices cannot be maintained. Prices will definitely fall as authors take control and cut out the middle man.  Those publishers who wish to continue in existence will have to radically rethink their processes to reduce overheads dramatically otherwise they will simply be unable to compete.

Part 1: Are textbooks obsolete

Part 2: E-textbooks

Part 3: Textbook publishing

Part 4: Author-led publishing

4 Comments

Filed under Uncategorized

The future of textbooks. Part 3: Textbook publishing

Textbook publishers don’t have a good reputation. They are seen to be avaricious in setting high prices for textbooks, exploitative in bringing out unwanted new editions and (by some authors) uncommunicative and unhelpful.

Certainly, textbook prices have risen at a significantly higher rate than inflation in general and they are unaffordable by many students. This results in fewer sales and so publishers raise prices to maintain their revenue – thus perpetuating the vicious circle.

Yet, in reality – textbook publishing is not outrageously profitable; authors don’t make lots of money from their books and bookstores are closing.  Something is seriously wrong in the system. It is fairly obvious to me that the publishing industry have to look at the way they do things and find ways of doing them differently and much more cheaply.

From an external perspective, it seems to me that part of the problem is that very little has fundamentally changed in book production and marketing in the last 20 years. Obviously manuscripts are no longer rekeyed and are emailed rather than posted to the publisher (although my most recent contract had a clause about submitting a double-spaced single sided copy on paper – which I ignored). Apart from that the processes in producing the 9th edition of my book were very little different from those in producing the 1st edition in 1982.

There is enormous scope here for more agile, technology-enabled processes that could reduce the time from delivery to the bookshop by at least 50%.  Instead of a set of sequential activities, creating a decent collaborative environment for book production could make everyone’s like easier. But, if we have a decent environment, authors can work directly with freelance editors, designers and typesetters. Recommendation systems mean that authors can tell each other about good people to work with and the publisher middleman can be dispensed with.

But what about the publisher’s role in quality assurance.  Getting a reputable publishers to produce and print your book means you’ll end up with a high-quality product. Wishful thinking. Some publishers who specialize in small print runs, such as Springer, ask authors to produce camera-ready pages then they never read what’s produced before printing. The quality is dreadful. Some publishers do employ designers but the general standard of textbook design (even books that sell thousands of copies) is so bad that it’s difficult to imagine that anyone with even a modicum of design literacy was involved. Not a great value add from publishers here.

Maybe the problem is printing, inventory management and distribution. These are certainly expensive but they will disappear when we move to e-books.  So, another, currently important function of publishers will disappear.

What does this leave – marketing and sales. Some publishers (e.g. Addison Wesley) make a decent effort here but, like all others, continue to delude themselves that people read email. I haven’t had a marketing email from publishers for years as my spam filters pick them up and junk them. Other publishers are just awful at marketing and seem to make no effort whatsoever.

Certainly, there’s a real need for marketing and sales and maybe this one reason that some publishers will survive.  But maybe more specialist marketing agencies, working with authors and using modern tools could do a better job.

So, is there a future for textbook publishers? Unless things change quite quickly and they take on board that the world has changed, I really doubt it.  Maybe a small number of publishers who focus on high-volume texts will survive – but I predict that most of the current textbook publishing companies won’t be around in 10 years time.

Part 1: Are textbooks obsolete

Part 2: E-textbooks

Part 3: Textbook publishing

Part 4: Author-led publishing

 

1 Comment

Filed under Uncategorized