I recently read a blog post on why papers were rejected for the ACM conference on Human Computer Interaction (CHI 2012). I found this to be profoundly depressing as it came out with a list of pseudo-scientific reasons why papers were rejected, without taking any account of the fact that demonstrating new ideas is not science.
I’m fairly sure that the author of the post was trying to be helpful and was well-intentioned but when these are the reasons for rejection, then it means it is highly unlikely that original ideas will be published. Imagine if Apple had submitted a paper on the ipod to this conference – no formal evaluation so not wanted! Rather, the ideal paper is an increment on something that has already been done and which is then analysed to death. Whether it is interesting or exciting doesn’t matter.
I’m not trying to get at the CHI community here – the situation is exactly the same in my area of software engineering. Papers for the ICSE conference are profoundly dull. In fact the only reason to attend this conference (and I guess CHI) is that there are lots of interesting workshops going on alongside them where people actually talk about ideas.
Contrast this situation with journal publication. I recently completed the 3rd revision of a paper that has been accepted for the CACM. I never bothered submitting this to any conference because it contained some controversial ideas and I knew that a reviewer would disagree with them and they were rejected. Not surprisingly, a reviewer of the journal paper also disagreed – but the result was not to recommend rejection but to ask for more details and clarification. Essentially, journal papers are refined through a conversation between the authors, reviewers and editors. And they are all the better for this.
Of course there is a place for papers that are incremental, with lots of analysis. That place is in journals where readers who are interested have time to read and digest the analysis. Conferences are where people get together and they should be exciting and should stimulate lots of discussions. This means their focus should be on ideas and originality, so that people argue about them and generate their own ideas.
But at the moment, if you have a new idea, don’t waste your time submitting it to a top CS conference. These have become vehicles for academics who want promotion or tenure and they have completely lost sight of what conferences should be about.
Thoughts on software and systems engineering 
I spent way to long in school (studying software engineering) and I had the exact same thoughts as you (about the same conferences too). Then someone pointed it out to me: We (researchers) are not ‘inventors’. Research is about asking a question and finding the answers to those questions. It was then that I realized research was not for me.
Now, should CHI or other SE conferences be targeted towards ‘inventors’ or ‘researchers’? I don’t know the answer to that.
Great post, BTW!
Well… I can’t see the difference between researchers and inventors. Answering a new question requires invention!
I think this is the essence when we are talking about an engineering discipline that builds things. The question is often how do we build X so that it is cheaper, faster, safer, etc. Or sometimes, how do we build something so that it’s simply more fun to use.
The difference comes in the ‘product’. In research, the product is an answer to a question, anything invented is just a means to an ends. For example, how do you build X so it’s faster/cheaper: you would have an answer to a question (Do Y and X will be faster by Z% and we’ve validated this within some % 19 times out of 20).
With ‘inventions’ the product is the invention. When I first had this discussion years ago, it was pointed out to me that Thomas Edison’s light bulb would have never made it into our publications because it’s not a ‘question’.
BTW, I’m not saying this is right or wrong, just how it was explained to me.
Ian Bull, I understand where you are coming from. In general, computer science is in a cross-dressing identity crisis. In software engineering, we don’t know if we’re engineers, scientists, or researchers and we’re wearing each other’s clothes. I offer these definitions: scientist: modeling reality, engineer: budgeting reality, researcher: stretching reality.
We must first understand what roles we need to play, then the venues for playing out those roles will naturally follow.
Dear Ian, I love your blog, but I find your critique in this post misplaced. First, my blog post was focused on flaws in *empirical work*. As it says in the beginning, it does not deal with other topics like originality. There’s absolutely nada about evaluation of originality in there. Secondly, I don’t agree that the reasons were pseudo-scientific. As many commenters have pointed out, you can find the same issues in textbooks on statistics and empirical methods. I do admit that when taken out of context (paper, reviews, balancing of pros and cons in the meta-review), the rationale look one-sided. However, instead of a sweeping claim about CHI being “pseudo-scientific”, it would be far more constructive if you explicated what kind of criteria is “scientific” and we should use. Obviously you know something we don’t. Thirdly, you’re wrong in implying that original ideas don’t get published at CHI. Anybody can have an idea. Few people have original ideas. Fewer can actually implement their idea. And even fewer can show convincing empirical evidence for them. And those ideas you find at CHI.
My own experience tells me that people don’t bet on ideas in general. People bet on people. It’s because all communities, including the scientific, is a social construct, where status and alliances matter. Thus, people rather bet on other people than on original ideas. Sad, yes, but we are humans.
Anttti – thank you for taking the time to comment. Your article simply prompted me to remark on the general problem with top conferences and CHI happened to be the subject – I think the problem is a universal one for all CS conferences with a high rejection rate.
In general, I am a proponent of empirical work (see this post) but this has to be properly published – in journals where there is scope for a dialogue with the authors and (critically IMHO) the possibility of providing an explanation of and links to the raw data. This is simply impossible in a short conference paper and it is unrealistic for a PC to expect this. What you will get will be pseudo scientific.
Some of your points were about replicability – which is seen as the holy grail of science. Actually, its much less common than we in CS imagine – we see the high profile replicable experiments but few others are. But it is frankly impossible when we are evaluating engineered artefacts – if I have built something, you don’t have it to replicate my experiments. So, I think it is again unrealistic to reject papers for this reason.
The essential problem is fundamental to the discipline – unlike most other disciplines, conference publication is favoured over journal publication so you get very high numbers of papers submitted. PCs have to look for reasons to reject and this means that original ideas, which are inevitably flawed, are rarely accepted. There may be original ideas published in CHI or ICSE but these are the exception rather than the norm.
Thank you Ian for your reply. If you agree that a key mission of of top conferences on an applied topic (e.g., HCI) is to publish *good ideas that are substantiated*, my blog points toward the threat that CHI has raised the bar too high for the evidence bit, neglecting the fact that it is hard to provide compelling evidence for a truly novel prototype. If this is the case, as some influential people working in interactive systems claim, then we’d see mainly mediocre-quality-ideas-with-solid-empiria type of papers published at CHI. This would be sad. But I argue that this is not the whole story. First, some people are able to do this feat, there are every year good ideas with good evidence published. Second, do we really know that there were good ideas among the rejected ones. We haven’t looked at evaluation of *novel ideas* in the meta-reviews (weighing of pros and cons), but simply reasons for rejections. To do this properly, maybe somebody could look at “the history of ideas” submitted to CHI, especially those ideas that turned out to be good later on, e.g. in the consumer market, and see how their CHI submissions (if any) were received. Third, I’d like to add that even if empirically-oriented subcommittees tend to require solid evidence, SCs and ACs get explicit instructions to favor novel ideas over boring-but-proven ideas. My hunch is that CHI is ready to assume more risk when dealing with promising new ideas than, say, the human factors community. However, we obviously can’t let in papers that have critical flaws. Fourth, the CHI for has submission categories that do not necessitate empirical evaluation (videos, interactivity, alt.chi).
In summary, I think there’s a fair chance of seeing truly original thinking at CHI. Of all conferences that I’ve attended, it’s the best in this respect. To return to your question, would we have seen iPod published at CHI? I think yes, but not as a full paper but as a interactivity demo or video. On the other hand, realistically, if an idea is ready for market, why publish it at CHI and let competitors know about it? Therefore, I wouldn’t expect to see good ideas at CHI that would be ready for market within 1-3 years. When it comes to novel ideas, we’re dealing with a perspective of 5-15 years.
Thanks for opening this discussion Ian, it has been useful (at least for me).
This is EXACTLY what I think!
The more a research community is divided into “religions”, the fewer original ideas get published in that community. That’s the case for CHI (sorry), less so for other CS areas (fortunately).
I don’t think you can generalize from “CHI” to “all top CS conferences.” CHI specifically focuses on usability experiments, with an intentional focus on methodology. If you have a completely novel idea that is not substantiated with experiments, then a full CHI paper is not the correct way to communicate your idea to others. There are other venues that will accept “hot” new ideas that haven’t been tested yet, as well as demos/posters/short papers at CHI.
As someone say on twitter: If this were social science or medicine, the Higgs boson would be well and truly “discovered” #p=0.05doesn’tmeanmuch.
You should submit to ACM SIGGRAPH. New ideas on unfinished full papers are more than welcome. I had 3 already on this style at this conference.
Generalizations are always a poor point of view. I would say that “Original ideas can’t ALWAYS get published in top conferences”, or, “Original ideas CAN be published in top conferences if you align your text presentation with the conference reviewing process”. Both parts are interested in the publication: you want to spread your idea formally in the scientific community (don’t you?); while the conference wants to select the best papers (not the best ideias). So, as you need them, learn how to dance their music first. Once I’ve heard of a teacher that science is not about new ideas, science is about new knowledge. And to create such knowledge, we must to rely on boring stuff that guarantees some essential methodology checks. In experimental works for example, reproducibility is essential. But I’ll not pretend that these checks are the only reason why papers are rejected. In fact there are a lot of politics involved in these conferences. If you submit a paper, you are trying to be part of a group – usually a community of academics – and they decide what should and shouldn’t be accepted, even though this decision is usually carefully taken trough a blind review process. So, in many cases, you can have a perfect paper submitted but rejected. It happens! A tip: if you have an original cool idea, write a non-fiction book and go speech at pop-sci events!
This is a nice discussion, but I think it should have started by a clear definition of “original”. Oh no, I’m kidding 😉 We all know that there is no definition for that and that this discussion is therefore pointless.
By the way, the second point I ddon’t see in this discussion is that conferences sell two things: (1) interesting discussions/papers in domain (and originality is part of the criteria pro establishing that) and (2) scarcity.
Yes, CHI rejected 1200 papers in 2012. There may be many scientific reasons for that, but I also suppose that they didn’t want to accept more than 200 papers this year, and that is why they had to reject 1200 papers. To be really “scientific”, the discussion in that blog post should have compared the feedback on rejected papers with the feedback on LOW SCORING ACCEPTED PAPERS.
Maybe I’m wrong, but I suppose that many of those papers would present some of the critical comments we see in the rejected ones.
I work in a good CS department within a Graduate School of Engineering. Next door are the physics and chemistry buildings.
It took me many years to understand what are the diffrerences between Engineering (the good ones who do research), natural Sciences and the so called Computer Science.
To make it short:
The Researchers in Natural Science DO NOT INVENT THINGS! They just discover, prove, and demonstrate what Nature puts there. Of course to do so they must be creative, but do not confuse the subject with the verb – the object of their findings was there already. Some of these people get credit for developing clever ideas on how they discovered, and prove those FINDINGS.
The Engineering researchers apply or develop methods for doing things cheaper, faster better, and sometimes apply those methods to prove them, mas also to obtain the results/data that will be useful for the next ones.
The CS people INVENT things. We have to invent new things all the time. They were not there fist hand. Nobody put those ideas or artifacts there. Worst, they will be true, provable and perhaps useful for a limited time, until a new paradigm change everything.
We should not confuse that field of study with the mathematics of computing and alike, that in Mathematics and goes in the first group above.
Today it would be more important the paper proving why a bicycle does not fall them a paper showing an amazing new vehicle that runs on two wheels and does not fall !
I have a new idea and found that no one was interested. Right or wrong there is only silence. I’m an engineer that has programmed in Ada so I have a lot of respect for programmers. I’ve done System Engineering as well as building a lot of electronic black boxes and boards. This dream of connecting the sciences has been with me a long time and had to wait until I retired.