Problems with peer review, Part Three

The problems I noted with peer review in my post below are, perhaps, too well-known to most people in not just the sciences, but the humanities as well. Too many submissions, not enough time, too much deference to authority, not enough recognition of new, potentially paradigm-challenging work, and so on. As a result, too many papers of poor quality get published, too many papers that could make interesting contributions to a field don't get published. The problem with this picture is that all the review is anonymous, and that a very small set of reviewers is in charge (over the community as a whole quite a few members might be reviewing, but for a given paper, the number is very small, sometimes just one or two, and there is no opportunity for author response or clarification).

Instead, it might be better for both workshops and conferences for the submitted papers to be made able to a wider audience, and with reviewers able to make both anonymous and identifiable comments, and authors able to make responses. Take a small workshop with approximately 10 members in its program committee. It receives something like 20-30 contributions, and it plans to select about 10 for final presentation. The submitted papers could be placed online with annotation tools for commenting and author responses. The program committee could send out notifications of submissions to the community at large, inviting commentary on the papers. The review period could begin on a rolling basis, with papers becoming available for review as they are submitted, and staying online till some point, at which stage authors could either submit a revised version or the PC could declare an end to the reviewing process.

One particularly salutary consequence of this system would be the chance for authors to respond to critiques, and for a real discussion to break out on the papers. In fact, I suggest that this process of public submission, open review, author response, ensuing discussion, and subsequent revisions might even be more valuable than the intended presentation at the workshop. For what happens in that scenario under the current system? The author sends the paper out, it gets reviewed in the hurried fashion I described in my last post, and then when it is submitted, his only audience is the small one at the workshop or conference (yes, there is the advantage of the oral presentation, but it can also very easily become a disadvantage). The community is only partially represented at the workshop (even if it is a specialized area, very few people can actually travel to workshops and conferences; funding is especially hard to come by for people in theoretical fields). The public process also ensures that poorly written, superficial, content-free papers that make it just because a PC member wrote a hasty, superficial review that was not vetted by anyone else on the PC will have a harder time getting through. It will also ensure that papers written by so-called 'authorities' will be subject to a wider critique than just a couple of possibly star-struck reviewers. And the length of the review process will also ensure more thoughtful reviews as people can add points over a period of time (I'd certainly consider making incremental comments on papers made available for public reviewing). But what about star-struck reviewers afraid of upsetting the 'stars' in the field? There isn't much that can be done if you are worried about reprisals but you needn't worry if your primary concern is that you might say something wrong. But why would this worry about reprisals be a problem? It'll only be one if a critique is made in intemperate fashion, where the content is obscured by its form. Which of course is a huge problem in anonymous reviewing where the amount of vituperative swiping from the safety of the anonymous reviewers position is quite amazing.

To sum up (and to be sure this is a very quick take on things), while the problem of volume in today's academic world can't be easily solved, the workshop and conference world in the sciences would greatly benefit from a public, open, extended, iterative review process. Plenty more to be said here, of course. but all in good time.