I have written in an earlier post about my concerns that the research community is being driven by targets to publish work that clearly isn’t ready for publication. I made the point that papers are submitted to conferences that don’t contain evaluations of the work and papers that are supposedly about software systems but where the systems have not actually been implemented.
Well – I had the unhappy experience today of reviewing conference papers (not HCI this time) on agile methods and software engineering – I reviewed 5 papers and not 1 had any information about evaluation. I am guessing that most of these papers were written by PhD students and that they felt compelled by the prevailing publication culture to submit papers to conferences of work in progress. This is really utter nonsense. Sometimes PhD students produce solid publishable work during their time as a student and sometimes they don’t. I have supervised both kinds of student and one is not better than another. It may make more sense to write a single, in-depth paper at the end of a 3 or 4 year period rather than a series of shorter papers.
But the people to blame here are the student’s supervisors or advisors (who are sometimes named on the papers). They should not be encouraging the submission of unfinished and premature work. They should be making absolutely clear to students that papers about vapourware or papers where there is no evaluation or comparison of the work with other approaches are simply not good enough.
There is also a need for organisers of conferences to make clear that papers that propose some practical approach and that do not include a discussion of evaluation will be rejected without review.And they should screen papers before sending them out for review – wasting reviewers time means that we will be less inclined to do reviews in future. If this means fewer paper submissions and so fewer conferences, this would be good for everyone concerned.
Thoughts on software and systems engineering 