On OOPSLA 2010's acceptance policy

The opening pages of a Proceedings volume, the editor's forward, is something one normally skips without even noticing. Yet the OOPSLA 2010 proceeding start with a rather unusual "Program Chair Statement" that is well worth reading. And debating.

The chair, Martin Rinard, joins the heated discussion that is raging on the pages of Communications of the ACM, under the editorial guidance of Moshe Y. Vardi, for quite some time now: how to improve the state of publications in computer science. See, for example, Vardi's editorial from the March 2010 issue: Revisiting the Publication Culture in Computing Research.

Rinard, however, decided to take action on the matter. And he introduced several key changes to the acceptance procedure for OOPSLA. These include:

• Unilateral paper acceptance, where each program committee member can pick a single paper that will be accepted, regardless of the opinions of the rest of the committee.


• Standard scientific review criteria, focusing on whether the paper introduces new results or techniques, are its results significant and interesting, and whether it contains errors.


• Reproducibility, measuring whether the paper contained sufficient information for other researchers to reproduce the result.

There were additional, more technical changes (e.g., with regard to submissions by program committee members, page limits, etc.).

Overall, I think these changes are most welcome. I particularly applaud the reproducibility concern, which (finally!) states the obvious: our field has a problem with papers by researches in the industry that work with proprietary systems, and produce papers that, while often fascinating, do not provide sufficient detail to truly advance the science.

It is the unilateral acceptance policy that bothers me; more specifically, the way it was executed. The cause, I should state, is a worthy one: as Rinard claims, "Papers that open up new directions or go against the established value system in the field are routinely rejected, either because they are controversial or because the program committee finds it difficult to evaluate the paper with the same degree of certainty as more mainstream papers"; and, in Rinard's view, "this situation has become severe enough to seriously hamper the development of the field" (!). In other words, the mundane wins over the challenging, and researchers prefer churning out mundane papers over doing ground-breaking research.

And is unilateral acceptance a good solution? I actually think it is. Except for this: Rinard chose to publish exactly which papers were unilaterally accepted, thus singling them out as the "odd ones"; worse still, the table of contents explicitly states which program committee member chose what paper.

The problem, obviously, is that this could create a sense of personal debt in the industry. Let's say committee member X picked a paper by author Y. Next year, Y would be in a committee of some other conference, to which X will submit a paper. Can we truly say that Y will never give X preferential treatment? Can we truly say that X will not hold a grudge against Y if he learns that, for this new conference, Y picked some other paper as part of his "unilateral acceptance prerogative" rather than picking X's?

Even if X and Y are both perfectly honest people, things can still seem wrong to others. If Y does pick X's paper in the following conference, will everyone believe it was a purely impersonal choice? When Y argues for X's paper in the next conference's PC meetings, how will the other PC members hold his views?

With the best of intentions, Rinard and the OOPSLA 2010 committee created a dangerous precedence. Yes, PC members should be able to pick one paper unilaterally. But they should justify their choice clearly to other PC members: why do they believe this paper they've picked is ground-breaking and novel. And more importantly, once the choice is made, the details should never be published in public; much like the scores given by PC members to different papers, during the review process, are never published except anonymously.

[Full disclosure: I have not submitted a paper to OOPSLA 2010. For that matter, since moving to the industry, I haven't submitted a paper to a conference in the past three years or so. So this isn't about me. I did take part in reviewing papers for several academic conferences during this time, though.]