Monday, February 18, 2008

Problems with peer review, Part Three

The problems I noted with peer review in my post below are, perhaps, too well-known to most people in not just the sciences, but the humanities as well. Too many submissions, not enough time, too much deference to authority, not enough recognition of new, potentially paradigm-challenging work, and so on. As a result, too many papers of poor quality get published, too many papers that could make interesting contributions to a field don't get published. The problem with this picture is that all the review is anonymous, and that a very small set of reviewers is in charge (over the community as a whole quite a few members might be reviewing, but for a given paper, the number is very small, sometimes just one or two, and there is no opportunity for author response or clarification).

Instead, it might be better for both workshops and conferences for the submitted papers to be made able to a wider audience, and with reviewers able to make both anonymous and identifiable comments, and authors able to make responses. Take a small workshop with approximately 10 members in its program committee. It receives something like 20-30 contributions, and it plans to select about 10 for final presentation. The submitted papers could be placed online with annotation tools for commenting and author responses. The program committee could send out notifications of submissions to the community at large, inviting commentary on the papers. The review period could begin on a rolling basis, with papers becoming available for review as they are submitted, and staying online till some point, at which stage authors could either submit a revised version or the PC could declare an end to the reviewing process.

One particularly salutary consequence of this system would be the chance for authors to respond to critiques, and for a real discussion to break out on the papers. In fact, I suggest that this process of public submission, open review, author response, ensuing discussion, and subsequent revisions might even be more valuable than the intended presentation at the workshop. For what happens in that scenario under the current system? The author sends the paper out, it gets reviewed in the hurried fashion I described in my last post, and then when it is submitted, his only audience is the small one at the workshop or conference (yes, there is the advantage of the oral presentation, but it can also very easily become a disadvantage). The community is only partially represented at the workshop (even if it is a specialized area, very few people can actually travel to workshops and conferences; funding is especially hard to come by for people in theoretical fields). The public process also ensures that poorly written, superficial, content-free papers that make it just because a PC member wrote a hasty, superficial review that was not vetted by anyone else on the PC will have a harder time getting through. It will also ensure that papers written by so-called 'authorities' will be subject to a wider critique than just a couple of possibly star-struck reviewers. And the length of the review process will also ensure more thoughtful reviews as people can add points over a period of time (I'd certainly consider making incremental comments on papers made available for public reviewing). But what about star-struck reviewers afraid of upsetting the 'stars' in the field? There isn't much that can be done if you are worried about reprisals but you needn't worry if your primary concern is that you might say something wrong. But why would this worry about reprisals be a problem? It'll only be one if a critique is made in intemperate fashion, where the content is obscured by its form. Which of course is a huge problem in anonymous reviewing where the amount of vituperative swiping from the safety of the anonymous reviewers position is quite amazing.

To sum up (and to be sure this is a very quick take on things), while the problem of volume in today's academic world can't be easily solved, the workshop and conference world in the sciences would greatly benefit from a public, open, extended, iterative review process. Plenty more to be said here, of course. but all in good time.

4 Comments:

Blogger M Butcher said...

Today I noticed something interesting: It took me eight months to write a 400 page book, work with the editors and reviewers on revisions, and get the work out the door.

It took ten months for a journal to which I submitted a twenty page article to get around to notifying me that the paper needed revision. I'm not even sure one of the reviewers actually read the paper (the comments were mostly tangental). And if I make the revisions, will it be another ten months? The article is barely relevant now!

Journals don't pay anything, while the book has had at least some payoff. The book has already sold more copies than the journal's circulation. So why should I write a journal article? Is "notoriety amongst one's peers" really that important? And is respect really gained primarily through journals?

Peer review is definitely part of the problem... but perhaps the idea of preserving "the journal" is a mistake altogether.

2:31 AM  
Blogger Samir Chopra said...

Hey Matt, good to see you here. Congrats on the book; you're a machine!

I agree with the point about journals. I should post about this real soon, but in a nutshell, I think the system is broke there too (I know I had said differently in an earlier post but your post reminded me of why I'm wrong). I think the only way to go is to have a time-bound, public, open review, after which the editors take it offline and ask the authors to revise. If the revisions don't work, or can't be done, the paper is done as well.

6:36 AM  
Anonymous Anonymous said...

I recently discovered the Code4Lib Journal, and their review process is quite a bit like the one you propose:

http://journal.code4lib.org/process-and-structure
(scroll down to "Submission")

But now I have a dilemma: I like Code4Lib's approach (and the fact that it's open access, of course), and I'm soon to write an article that would be well suited for it. But I'm pre-tenure, and I don't know if I can afford to publish in a journal that isn't peer-reviewed in the traditional, tenure-committee-pleasing way. I may well decide to submit to Code4Lib, but if I do, it's a bit of a professional risk.

So problem #1 is changing peer review. And problem #2 is broadening acceptance (in high-stakes situations) for this new kind of peer review.

1:03 PM  
Anonymous generic cialis 20mg said...

Hello, I do not agree with the previous commentator - not so simple

3:37 AM  

Post a Comment

<< Home