Personal responsibility and research funding

I was recently asked to review a research proposal where the proposer, who I know and respect, is  past the ‘normal’ retirement age. He has a distinguished track record of research and the work proposed was very good. It was a high quality proposal and I supported it.

But I think it was highly irresponsible of the applicant to submit this proposal. It may be good for science but it was bad for the research community. Why? Because the old guys are getting more than their share of the money and this means that we are making it more and more difficult for young researchers to get their first step on the ladder.

It might be argued that research funding bodies make dispassionate decisions based on the quality of the research that is proposed and if young researchers have the best ideas, then these will get funded. This is arrant nonsense. Not only is proposal writing a skill in itself, which takes time and experience to learn, researchers with an established track record get an easier ride. This makes complete sense – if people have done good work in the past, you can expect them to do good work in the future. Consequently, reviewers overlook issues in a proposal that they would be otherwise be concerned about.

Therefore, if you have no track record and are starting out on your research career it is harder to get funding. But it is absolutely essential that we (the research community) provide a route for early career researchers to develop as independent research scientists and engineers.

The best way to do this, of course, would be for research funders to make a policy decision that no-one over the age of 60 (say) can submit a new application for funding as a principal investigator. But, age discrimination is not allowed so this is impossible.

But, we can do this voluntarily – instead of submitting proposals as principals, we old guys should step back and help our younger colleagues rather than competing with them. Let us use our expertise to advise the next generation instead of creating a situation where many of them will be so disillusioned that they will simply give up research. We don’t need the money and, if we are good enough to get proposals funded, we’ve already established our reputation.

It will be argued that this might mean that the ‘best’ science is not supported. Again, arrant nonsense. Research ought to be risky which means that what is ‘the best research’ cannot be decided objectively. The research councils reject lots of high quality proposals and often the decision on where a proposal is ranked is simply an accident of reviewing.

Personally, I have about 30 years experience of writing successful research proposals so I think that I can speak with some authority here. I don’t intend to hang up my researcher hat yet but I will not submit any new research proposals as a principal applicant to the EPSRC or to EU research programmes.

I know that in an ideal world, this would not be necessary as there would be sufficient funding for everyone. But waiting for an ideal world has never been a very practical strategy.

Leave a comment

Filed under research

Pi in the Sky?

The remarkable low-cost Raspberry Pi computer went on sale today, and has apparently sold out in two hours. The Raspberry Pi is a very cheap computing device that can be the basis for do-it-yourself systems and it has been suggested that it will revolutionise the teaching of computer science in schools. Instead of boring IT, students will experience the thrill of making something themselves and will, consequently, be inspired into further studies of computer science.

This is a great development, which I wholeheartedly support – computer science teaching in schools has left a great deal to be desired. And, for sure, it is possible that it will enthuse technically-oriented students to find out more about computer science and programming. It’ll be fun and I’m sure we’ll see lots of innovative and exciting development based on this device.

Now I really don’t want to rain on the Pi parade but, I find it hard to believe that this device will have much success in convincing the roughly 50% of school students who have, up till now, shown very little interest in any kind of computer science – girls.

As the father of two daughters, I have watched them develop in a technically literate household where they had access to computers and programming advice if they wanted it. But, their view and the view of all of their friends that I talked with was that programming was for geeks and nerds (I am not sure of the difference between these). While they were perfectly competent in using computers for various tasks, they  showed no interest whatsoever in programming as they simply did not see how it could be useful to them. The social image of programmers, of course, did not help – teenagers are very image conscious and being associated with an uncool group was something to be avoided.

If anything, the Raspberry Pi may make this situation worse, especially if it is taken over by teenage boys in a class who, shall we say, may not have the best social skills in the world.  Yes, it will re-introduce programming into the curriculum but that is only one problem that we have to face. The much more difficult challenge is to be able to demonstrate the relevance of programming to the whole population – not the technically-minded subset that will surely love their PIs. This won’t happen from bottom-up  cheap devices, but will need a top-down, application-oriented approach.

I was impressed by the efforts made in Georgia Tech to try and broaden the CS curriculum and demonstrate the relevance of programming to a wide spectrum of students. I don’t know how well these have succeeded in attracting more girls to computer science but it strikes me that this is the kind of initiative we need in schools to demonstrate the relevance of computer science to girls.

Doing this is hard – I certainly don’t have an answer to the problem of making CS female-friendly. But I do have concerns that the Raspberry Pi will simply reinforce the stereotype of computer science being a geeky subject that has nothing to offer to girls.

Leave a comment

Filed under CS education

Risk-driven approaches to dependability

I’ve been reading a lot about cloud security today as, perhaps rather hastily, I offered to lead a discussion on my gut feeling that there is really nothing new in cloud security.  When you read articles on this topic, what strikes you is that they focus on security technicalities rather than the security risks that businesses face every day. I’ve written about the specific issues around cloud security in my Cloudscape blog.

But this brings me to a more general point that I make in my book but which perhaps needs emphasising again. When you have a limited amount of resources to spend on achieving dependability, start by identifying the risks and threats to system dependability. Focus on those risks which have a (relatively) high probability of occurring and the risks that have serious consequences.  Think about how your software and your testing process should cope with these problems – if you can avoid the biggies, then you will achieve dependability.

This is one of the problems that I have with automated testing. There is an emphasis on taking bottom-up approach, where you write unit tests for a component, with no idea of whether these include practical usage scenarios. There is a tendency to think that software that passes all the automated tests is necessarily dependable – but if you haven’t covered all the risks, then you could be in for a surprise.

Leave a comment

Filed under dependability, software engineering

The IEEE simply doesn’t get open access publishing

I think that open-access publishing where research papers are freely accessible on the web is ultimately inevitably although it will take a while to break the monopoly of the big academic publishers (see this article by George Monbiot to see how bad this is).  Ethically, if researchers are paid to do research through public funds, then the public have a right to access their work.

Organising publication and peer reviews is not free, however, so a fee has to be paid by the author. This is a good thing in my view as it means that authors will think twice before submitting work that is not mature enough and may ultimately reduce the total number of papers published.

Now most journals as run by volunteers so, typically the editor and reviewers are not paid. So the principal cost is perhaps some administrative support for the editor.  If we assume that the cost of a full-time administrator + office space/phone/computer is about £60, 000 per year and a journal publishes 10 papers per month for 12 months, then the publication cost should be around £500. Details don’t matter – the essence is that the costs are hundreds not thousands of pounds/dollars/euros. 

The IEEE recently announced that it would provide what they call ‘hybrid open access’ where authors can opt for their papers to be openly available, at a cost of $3000.  However, the other papers in the journal are still only available to subscribers and, of course, there are no signs of the subscriptions being reduced.  If $3000 is the real cost (taking into account that advertising and subscriptions also contribute) then it looks to me like the IEEE is a pretty inefficient organization.

But the point here is that hybrid open access is complete nonsense. Either a journal is open access – available freely to all or it isn’t, There is no halfway house that makes sense – its unfair on authors to charge them as if there were no subscribers and its unfair on subscribers not to reduce their subscription if some articles are open-access. 

We see more evidence that the IEEE doesn’t understand the model (or, more likely, is doing all it can to stop open-access publishing) in its idiotic statement about additional page costs

“Overlength page charges support the costs of printing extra pages above a standard length set by the journal”

Open access is NOT about print publishing – printed journals are an anachronism and I’m willing to bet that the majority of IEEE journal users don’t get printed versions. 

I’ve been a member of the IEEE for almost 30 years and I like their journals. I think that professional societies should be the leaders in promoting open access and I’m saddened by the fact that either they don’t understand this or that they wish to discourage this approach to publishing.

Personally, I’ve recently taken the position that I won’t review for journals that don’t make articles freely available and I won’t submit articles to them. See:

http://www.researchwithoutwalls.org/

2 Comments

Filed under Uncategorized

Original ideas don’t get published in top conferences

I recently read a blog post on why papers were rejected for the ACM conference on Human Computer Interaction (CHI 2012). I found this to be profoundly depressing as it came out with a list of pseudo-scientific reasons why papers were rejected, without taking any account of the fact that demonstrating new ideas is not science.

I’m fairly sure that the author of the post was trying to be helpful and was well-intentioned but when these are the reasons for rejection, then it means it is highly unlikely that original ideas will be published. Imagine if Apple had submitted a paper on the ipod to this conference – no formal evaluation so not wanted!  Rather, the ideal paper is an increment on something that has already been done and which is then analysed to death. Whether it is interesting or exciting doesn’t matter.

I’m not trying to get at the CHI community here – the situation is exactly the same in my area of software engineering. Papers for the ICSE conference are profoundly dull. In fact the only reason to attend this conference (and I guess CHI) is that there are lots of interesting workshops going on alongside them where people actually talk about ideas.

Contrast this situation with journal publication. I recently completed the 3rd revision of a paper that has been accepted for the CACM. I never bothered submitting this to any conference because it contained some controversial ideas and I knew that a reviewer would disagree with them and they were rejected. Not surprisingly, a reviewer of the journal paper also disagreed – but the result was not to recommend rejection but to ask for more details and clarification. Essentially, journal papers are refined through a conversation between the authors, reviewers and editors. And they are all the better for this.

Of course there is a place for papers that are incremental, with lots of analysis. That place is in journals where readers who are interested have time to read and digest the analysis. Conferences are where people get together and they should be exciting and should stimulate lots of discussions. This means their focus should be on ideas and originality, so that people argue about them and generate their own ideas.

But at the moment, if you have a new idea, don’t waste your time submitting it to a top CS conference. These have become vehicles for academics who want promotion or tenure and they have completely lost sight of what conferences should be about.

18 Comments

Filed under Uncategorized

A research agenda for software engineering

I have written elsewhere in this blog on how I believe that reductionism as a basis for software engineering is not appropriate for the complex coalitions of systems that we are now building. This means that ‘more of the same’ – improving existing software engineering methods – simply will not allow us to tackle the development of the large-scale, complex software systems that are rapidly becoming a reality.

Rather, I believe that we need to completely rethink software engineering research and propose the following top-10 research problems that I think need to be addressed:

1.      How can we model and simulate the interactions between independent systems?
To help us understand and manage coalitions of systems we need dynamic models that are updated in real-time with information from the actual system. We need these models to help us make rapid ‘what-if’ assessments of the consequences of system change options. This will require new performance and failure modelling techniques where the models can adapt automatically from system monitoring data.

2.      How can we monitor coalitions of systems and what are the warning signs of problems?

To help avoid the transitions to sunstable system state, we need to know what are the indicators that provide information about the state of the coalition of systems, how these indicators may be used to provide both early warnings of system problems and, if necessary, switch to safe-mode operating conditions that will stop damage occurring. To make effective use of this data, we need visualization techniques that reveal the subtleties of coalition operation and interactions to operators and users.

3.      How can systems be designed to recover from failure?

As we construct coalitions of systems with independently-managed elements and negotiated requirements, it is increasingly impractical to avoid ‘failure’. Indeed, what seems to be a ‘failure’ for some users may not affect some others.  Because some failures are ambiguous, automated systems cannot cope on their own. Human operators have to use information from the system and intervene to recover from the failure and restore the system. This means that we need to understand the socio-technical processes of failure recovery, the support that these operators need and how to design coalition members to be ‘good citizens’ and to support failure recovery.

4.      How can we integrate socio-technical factors into systems and software engineering methods?

Software and systems engineering methods have been created to support the development of technical systems and, by and large, consider human, social and organisational issues to be outside the system boundary. However, these non-technical factors significantly affect the development, integration and operation of coalitions of systems. There is a considerable body of work on socio-technical systems but this has not been ‘industrialised’ and made accessible to practitioners.

5.      To what extent can coalitions of systems be self-managing?

The coalitions of systems that will be created are complex and dynamic and it will be difficult to keep track of system operation and respond in a timely way to the monitoring and health measurement information that is provided. We need research into self-management so that systems can detect changes in both their own operation and in their operational environment and dynamically reconfigure themselves to cope with these changes. The danger is that reconfiguration will create further problems so a key requirement is for these techniques to operate in a safe, predictable and auditable way and to ensure that self-management does not conflict with ‘design for recovery’.

6.      How can we manage complex, dynamically changing system configurations?

Coalitions of systems will be constructed by orchestration and configuration and the desired system configurations will change dynamically in response to load, indicators of the system health, unavailability of components and system health warnings. We need ways of supporting construction by configuration, managing configuration changes and recording changes (including automated changes from the self-management system) in real-time so that we have an audit trail recording what the configuration of the coalition was at any point in time.

7.      How can we support the agile engineering of coalitions of systems?

The business environment changes incredibly quickly in response to economic circumstances, competition and business reorganization. The coalitions of systems that we create will have to change rapidly to reflect new business needs. A model of system change that relies on lengthy processes of requirements analysis and approval simply will not work.

Agile methods of programming have been successful for small to medium sized systems where the dominant activity is system development. For large and complex systems, development processes are often dominated by coordination activities involving multiple stakeholders and engineers who are not co-located. How can we evolve agile approaches that are effective for ‘systems development in the small’ to support multi-organization, global software development?

8.      How should coalitions of systems be regulated and certified?

Many coalitions of systems will be critical systems whose failure could threaten individuals, organizations and economies. They may have to be certified by a regulator who will check that, as far as possible, the systems will not pose a threat to their operators or the wider systems’ environment. But certification is increasingly expensive. For some safety-critical systems the cost of certification may exceed the costs of development. These costs will continue to rise as systems become larger and more complex.

9.      How can we do ‘probabilistic verification’ of systems?

Our current techniques of system testing and more formal analysis are based on the assumption that the system has a definitive specification and that behaviour which deviates from that specification can be recognized. Coalitions of systems will have no such specification nor will system behaviour be guaranteed to be deterministic. The key verification issue will not be ‘is the system correct’ but ‘what is the probability that it satisfies essential properties, such as safety, that take into account its probabilistic, real-time and non-deterministic behaviour’.

10.    How should shared knowledge in a coalition of systems be represented?

We assume that the systems in a coalition will interact through service interfaces so there will not be any over-arching controller in the system. Information will be encoded in a standards-based representation.  The key problem will not therefore be a problem of compatibility – it will be a problem of understanding what the information that systems exchange actually means.

Currently, we address this problem on a system by system basis with negotiations taking place between system owners to clarify what shared information means. However, if we allow for dynamic coalitions with systems entering and leaving the coalition, this is no longer a practical approach.  The key issue is developing a means of sharing the meaning of information – perhaps using ontologies as proposed in the work on the semantic web.

Thanks to Dave Cliff and Radu Calinescu for their input. More on this in our paper on Large Scale Complex IT Systems.

1 Comment

Filed under Uncategorized

Science or Snake Oil: Empirical software engineering

In my professional lifetime, I’ve seen the emergence, development and adoption into the mainstream of object-oriented development for software. Instead of functions or procedures being the principal unit of abstraction, objects that encapsulate data and operations are now seen as the ‘natural’ abstraction in programs, at least outside the embedded systems domain.   I would be surprised if any university computer science course didn’t teach OO programming and my guess is that students are rarely taught to think about whether or not objects really are the right abstraction.

Now, I think the idea of abstract data types, where data and its related functionality are encapsulated is a pretty natural one. It makes sense when thinking about how to organize some data, to associate the operations with that data. But I have never found thinking about a problem to be solved in terms of objects to be a natural approach. I think about what to do to solve the problem and, this seems to me to be a pretty general trait – we articulate solutions to problems in terms of actions.

The fact that objects are not a natural way of thinking does not mean that they are not a good way of programming. This is not really the point of this post – the point is that OO development has become almost universally accepted  in spite of the fact that virtually all empirical studies on its effectiveness show that OO development has no real advantages over a functional approach.

This paper, which studies development productivity in C and C++ makes the telling point:

We investigated whether C++ improved the productivity of software development. We did not find any empirical evidence of improved productivity.  This is in line with previous studies that to date have found no convincing empirical support in favor of object-oriented development.

The last sentence is the telling one – essentially, there is no science to back up the adoption of OO methods and techniques.

Of course, this does not mean that object-oriented approaches are ineffective – they are obviously useful and usable for large system development. But, we have to ask ourselves if this effectiveness is inherent in the approach or because there has been a vast amount of effort and money invested in the development of tools and methods to support the object-oriented approach. I wonder if we had invested the same amount of money and effort in developing tools and methods for languages such as Modula 2  instead of Java  would we have a better programming environment?

The point of this post is not really about OO development, however. What I find concerning is that empirical  work in software engineering is essentially ignored by almost everyone. The paper I’ve linked above was published in 2008 but Les Hatton wrote a paper  in 1996 about studies of OO development and there may well have been others before that.

One problem is our academic publishing system where interesting results are published in obscure conferences and journals that no-one reads and authors get no academic credit for making their material more accessible to a wide audience. Another problem is our research funding system that focuses on novelty rather than science – it’s very hard to get funding for a project that aims to confirm or deny previous work.

But the biggest problem, I think, are the snake oil salesmen. The sellers of universal solutions who don’t admit there is no scientific basis for the claims they are making, who don’t admit there are some areas where their techniques are inapplicable and who are selective with the truth when discussing the use of their techniques.

We are now seeing another ‘revolution’ in software engineering as the agile methods bandwagon becomes unstoppable. I have a lot of time for some aspects of agile methods but we should be honest about their effectiveness and applicability.

For example, few discussions of test-driven development, a central agile development technique, admit that empirical studies of TDD (in industry (Microsoft and IBM) show that it slows down development – so you might think it is less rather than more agile. These studies also show that it leads to fewer post-release problems, confirming the technique is of value but not, perhaps, for the reasons given by the agile salesmen.

Sadly, there are too many vested interests here – academics seeking tenure and promotion and consultants seeking contracts – that a change to a more scientific approach in the near future is unlikely. The people who will have to pay the price will be the adopters of these techniques who will learn that the magic snake oil is not a solution to their problems after all.

10 Comments

Filed under Uncategorized