Tag Archives: research

Time for a harder line on evaluation

I have written in an earlier post about my concerns that the research community is being driven by targets to publish work that clearly isn’t ready for publication. I made the point that papers are submitted to conferences that don’t contain evaluations of the work and papers that are supposedly about software systems but where the systems have not actually been implemented.

Well – I had the unhappy experience today of reviewing conference papers (not HCI this time) on agile methods and software engineering – I reviewed 5 papers and not 1 had any information about evaluation. I am guessing that most of these papers were written by PhD students and that they felt compelled by the prevailing publication culture to submit papers to conferences of work in progress. This is really utter nonsense.  Sometimes PhD students produce solid publishable work during their time as a student and sometimes they don’t. I have supervised both kinds of student and one is not better than another. It may make more sense to write a single, in-depth paper at the end of a 3 or 4 year period rather than a series of shorter papers.

But the people to blame here are the student’s supervisors or advisors (who are sometimes named on the papers). They should not be encouraging the submission of unfinished and premature work. They should be making absolutely clear to students that papers about vapourware or papers where there is no evaluation or comparison of the work with other approaches are simply not good enough.

There is also a need for organisers of conferences to make clear that papers that propose some practical approach and that do not include a discussion of evaluation will be rejected without review.And they should screen papers before sending them out for review – wasting reviewers time means that we will be less inclined to do reviews in future.  If this means fewer paper submissions and so fewer conferences, this would be good for everyone concerned.

Advertisements

5 Comments

Filed under research, software engineering

CS publishing culture considered harmful

I read this article today (http://www.theatlantic.com/culture/archive/2010/07/whats-wrong-with-the-american-university-system/60458/) on what’s wrong with the US university system in between reading a bunch of papers on HCI. I’m pleased to say that I don’t think the teaching/research dichotomy is as bad in the UK and I think it’s pretty good in my own institution at St Andrews.

However, the sentence that really resonated was this one:

“The problem is that there are just too many publications and too many people publishing”

The papers I was reading today are certainly some of the ‘too many publications’ and we’ve got into this dreadful state because of our idiotic research culture. This is partly driven by tenure (in the US) and research assessment (in the UK) but, in truth, we have brought it on ourselves by favouring quantity over quality and by pretending that papers that report provisional and interim results are high-quality publications.

Most of the papers I read today were conference papers and, with one exception, they didn’t report on finished work. They reported on prototype designs, which were never evaluated for real systems or on system studies that were never followed up with the results used to modify the systems studied or used as a basis for developing some general theories that could be more generally applicable. These were HCI papers but it’s the same right across the discipline – there’s no shame in not implementing things, not evaluating work and not caring about its relevance to society.

Now I’m pretty confident that the authors of these papers are bright, thoughtful and creative researchers – they are publishing in the ‘best’ HCI conferences so it’s reasonable to think they are the top people in their field. They are smart enough to know how to play the system and are doing it well. But, the system is forcing them to always seek novelty instead of spending time doing long-term work and documenting it properly. The consequence is, sadly, mostly useless publications because they can’t really be used as a foundation for further work.

Conferences are great fun – you get to meet colleagues and friends from far away, have interesting tech talks, usually some good dinners. Hardly anyone listens to the papers though – either they are reading their email (imagine the uproar if you arranged a conference without wifi in the lecture rooms) or talking in the lobby. So lets start having (many) fewer conferences which are not about publication but which are about interaction. We need many fewer but much better and more complete papers.

Changing culture is always hard, always long term and usually needs some kind of major shock to initiate the change. Maybe the economic crisis is the shock that we need and will trigger a process that will lead to a grown-up CS research culture.

1 Comment

Filed under Uncategorized

Every cut has a silver lining

The UK Government has announced cuts of up to 25% which will be imposed on universities over the next 4 years.

I’ve been reading a recent document from IBM called Capitalising on Complexity, which emphasises the importance of innovation and creativity and this has triggered a reflection on the contribution that the computer science research community in universities can make to this. Sadly, the conclusion I’ve come to is “if we can do anything it is in spite of rather than because of existing research structures and management”.

The problems that we suffer from are primarily imposed by the need for research quality to be ‘measured’ – either at the individual level (career progression, tenure, etc.) or at the institutional level. We are all encouraged to publish regularly in ‘high-quality journals or conferences’ and to write research proposals for external research support. More and more people are now competing for very limited funding.

The end-result of this is conservatism and incrementalism. It is dangerous to your career to go into a new area or to think differently as there are no ‘high quality’ journals and conferences to publish in. If you make proposals where you suggest interesting questions to explore with no clear idea of the results you will achieve (what I think of as real research), you have zero chance of funding because your proposal will inevitably have lots of holes in it that reviewers can challenge.

Research funding bodies, to their credit, are aware of this problem and sometimes support special initiatives (like the LSCITS project) to try and be more innovative. By and large, however, these rarely work as the pressures for incrementalism that are imposed by the current university system are just too great. Researchers have to think of their future – if they take 3 or 5 years out to ‘think differently’, then they will probably never get another research job.

All of this means that CS research in universities is not the lever for innovation that it should be, it does not encourage creativity, nor is it addressing the grand societal challenges that we face.

Paradoxically, perhaps, the inevitable cuts in university and research funding may offer us a way out of this situation. If there are no research jobs, then the notion of a research career is less important and smart people don’t have to be so concerned about publications. Cuts in travel budgets mean that less time is spent travelling to conferences to present papers to people who are mostly reading their email anyway. The hateful research assessment may disappear and we can start thinking long term rather than writing about another incremental advance.  Maybe some of us oldsters will be kicked into early retirement before senility sets in and we will have time to think  differently.

But we must try and maintain support for our PhD students. PhD’s themselves are mostly incremental – students have to write and defend a thesis and innovation is inherently risky. But PhD students have time to think, to be innovative and to come up with new and exciting ideas for the future. With fewer research jobs, they may focus on startups who are, it seems to me, to be the true source of innovation nowadays.

Leave a comment

Filed under LSCITS