Problems with peer review, Part Three

The problems I noted with peer review in my post below are, perhaps, too well-known to most people in not just the sciences, but the humanities as well. Too many submissions, not enough time, too much deference to authority, not enough recognition of new, potentially paradigm-challenging work, and so on. As a result, too many papers of poor quality get published, too many papers that could make interesting contributions to a field don't get published. The problem with this picture is that all the review is anonymous, and that a very small set of reviewers is in charge (over the community as a whole quite a few members might be reviewing, but for a given paper, the number is very small, sometimes just one or two, and there is no opportunity for author response or clarification).

Instead, it might be better for both workshops and conferences for the submitted papers to be made able to a wider audience, and with reviewers able to make both anonymous and identifiable comments, and authors able to make responses. Take a small workshop with approximately 10 members in its program committee. It receives something like 20-30 contributions, and it plans to select about 10 for final presentation. The submitted papers could be placed online with annotation tools for commenting and author responses. The program committee could send out notifications of submissions to the community at large, inviting commentary on the papers. The review period could begin on a rolling basis, with papers becoming available for review as they are submitted, and staying online till some point, at which stage authors could either submit a revised version or the PC could declare an end to the reviewing process.

One particularly salutary consequence of this system would be the chance for authors to respond to critiques, and for a real discussion to break out on the papers. In fact, I suggest that this process of public submission, open review, author response, ensuing discussion, and subsequent revisions might even be more valuable than the intended presentation at the workshop. For what happens in that scenario under the current system? The author sends the paper out, it gets reviewed in the hurried fashion I described in my last post, and then when it is submitted, his only audience is the small one at the workshop or conference (yes, there is the advantage of the oral presentation, but it can also very easily become a disadvantage). The community is only partially represented at the workshop (even if it is a specialized area, very few people can actually travel to workshops and conferences; funding is especially hard to come by for people in theoretical fields). The public process also ensures that poorly written, superficial, content-free papers that make it just because a PC member wrote a hasty, superficial review that was not vetted by anyone else on the PC will have a harder time getting through. It will also ensure that papers written by so-called 'authorities' will be subject to a wider critique than just a couple of possibly star-struck reviewers. And the length of the review process will also ensure more thoughtful reviews as people can add points over a period of time (I'd certainly consider making incremental comments on papers made available for public reviewing). But what about star-struck reviewers afraid of upsetting the 'stars' in the field? There isn't much that can be done if you are worried about reprisals but you needn't worry if your primary concern is that you might say something wrong. But why would this worry about reprisals be a problem? It'll only be one if a critique is made in intemperate fashion, where the content is obscured by its form. Which of course is a huge problem in anonymous reviewing where the amount of vituperative swiping from the safety of the anonymous reviewers position is quite amazing.

To sum up (and to be sure this is a very quick take on things), while the problem of volume in today's academic world can't be easily solved, the workshop and conference world in the sciences would greatly benefit from a public, open, extended, iterative review process. Plenty more to be said here, of course. but all in good time.

Problems with peer review, Part Two

As usual, I begin with an apology. Sorry for taking so long to get to the second part of this intended series of posts on peer review. So, why do I think peer review is busted in the sciences? (please see caveats below about this being based on my own personal experience). Firstly, and I think this should come as no surprise to anyone, there is simply too much material being sent forward for publication. In computer science, there are thousands of annual conferences and workshops held annually. I don't exaggerate, you simply need to check whether on any given day there are at least three events being staged somewhere or not. The reviewing for these events is typically done by the program committee (PC), a bunch of academics who got together to organize the event; some of them were part of the original plan to put together the workshop or conference, others were invited to serve on the PC for various reasons (sometimes to add heft to the PC - as academics will often judge a meeting's quality by the star rating of the PC, and sometimes, quite simply, to aid in the reviewing). When submissions arrive, the papers are parceled out to the PC for reviewing. Sometimes papers are assigned to more than one member of the PC. More often than not, this stage of the reviewing is one-way-blind (I know the name of the author, but he does not know mine). In larger conferences, the reviewing is double-blind. More often than not, the PC member is over-committed. He has signed up for as many academic invitations as he can, all in a rush to add lines to the CV, to increase his visibility in the community, to network a bit more. But now, the papers are in the Inbox, and they need to be reviewed. Typically, the PC member is late with the reviews. He then receives reminder emails from the head of the PC, and he rushes off to review the paper, which is invariably read in perfunctory fashion, and then hastily reviewed/summarized/critiqued. The effect of this on the quality of the papers submitted to a typical event should be clear. Sometimes, the PC member will sub-contract the reviewing, either handing it on to a Ph.D student or to a colleague who he thinks might be able to help out (I should point out that Ph.D students can be both very harsh, or very mild, reviewers; the former is eager to show off his talents and knowledge, the latter is still convinced he does not belong in academia, and is very diffident in his reviews).

There are other problems. Sometimes a workshop or a conference will not receive enough submissions. Then the PC members panic; the event will not be viable if a miniscule number of papers are accepted. At this stage, other instructions go out to the PC members: "lets accept papers if they will spark discussion; lets accept them if they show some promise; lets accept them even if is not-met". So the event floats and all is well. The quality of the papers is uneven, but at least the workshop or conference did not get canceled.

There are problems of authority. Publications in premier conferences carry a great deal of prestige in the community. Paper acceptances are much desired. And attendance lists are familiar. Some of this has to do with the quality of the papers, some of this has to do with the established nature of the authors. Double-blind reviewing sounds very good in theory; but in fact, its quite easy to make out who the author of a paper is: writing style, subject matter, even the formatting style of mathematical symbols (a research group in France insisted on using MS-Word to format their papers, as opposed to Latex, others used idiosyncratic symbols for logical operators). A not-so-confident reviewer, confronted with a paper written by an 'authority', holds fire. The paper makes it through. Yet another, knowing that this is written by an 'authority', simply lets it go through, because 'it must be good'; others simply support friendly research groups. Peer review responsibility has been abdicated, and because a small group has been picked, there are no other opportunities to correct this. And often, because paradigms are jostling for first place (as often happened in my field, logics for artificial intelligence), reviewers are not too keen to promote papers that promote rival paradigms (but are keen to promote those that show their own favored paradigm in a good light). A colleague of mine who was trying to suggest an alternative formal framework had great difficulty getting his papers accepted; reviews of his paper were clearly off-base, prejudiced and hostile. Finally, another academic advised him to simply forget about the premier conferences and concentrate on journals whose editors would intervene, and who would guarantee him a chance to respond to his referees. So much for the impartiality of the peer review process.

Not much can be done about the volume of publication/writing problem. The modern academy demands that everyone get on the writing and publishing treadmill, and like obedient children, we jump on (how else would we get promotion and tenure?). But something can be done about the blind reviewing problem, all imperfect solutions to be sure, but they strike me as offering a better chance of ensuring the quality of that which gets through to be published. More on that later. I'll also try and write a bit on grant proposal review.

Problems with peer-review, Part One

In Chapter 4 (Free Software and the Scientific
Practice of Computer Science) of Decoding Liberation, while writing of the ideal of objectivity in science and the role of free software in ensuring it in computer science, we spent some time examining the phenomenon of peer-review. In this post (there'll be two actually), I want to revisit that discussion by way of amplifying one of the points made in there. I'll post a couple of small parts in this first post, and then follow-up tomorrow with my own comments

In Chapter 4, We started by noting that

Free software and current scientific practice share a reliance on peer review to help ensure that results are of the highest possible objective quality. Peer review’s role in science was formalized in the eighteenth century, when the Royal Society of London’s “Committee on Papers” was granted the power to “solicit expert opinions.” Peer review became an indispensable part of scientific practice due to the sharp increase in scientific work after the Second World War (Drummond 2003). Just as the increased complexity of science, due to its increasingly mathematical nature, required scientists to conduct peer review in the era of patronfunded science during the Renaissance, the increase in both variety of disciplines and volume of submissions drove formerly self-reliant journal editorial boards to seek larger pools of reviewers.

From this point onwards, though, we note a problem, which will ultimately be the subject of these posts:

But peer review, especially its anonymous variant, might not improve the rigor of the review process and thus not adequately facilitate objectivity (van Rooyen et al. 1999). Instead, anonymous peer review might act as a damper on innovation, by placing guardians at the gates to science: paradigms remain unchallenged as the authority of powerful scientists remains unquestioned (Horrobin 1990). The discipline of computer science is not immune to these problems; anecdotal evidence seems to suggest that practitioners are disgruntled about this process. Anonymous critique of papers, they point out, results in a familiar attendance list at premier academic conferences. But a more serious charge can be leveled against anonymous peer review: it provides no guarantee of the quality of published work (Horrobin 1990, 1996, 1981). An examination (Rothwell and Martyn 2000) of the correlation among reviewers’ evaluations of submissions to neuroscience journals and conferences revealed that

For one journal, the relationships among the reviewers’ opinions were no better than that obtained by chance. For the other journal, the relationship was only fractionally better. For the meeting abstracts, the content of the abstract accounted for only about 10 to 20 percent of the variance in opinion of referees, and other factors accounted for 80 to 90 percent of the variance. (Horrobin 2001)

It is difficult to value this form of peer review when little distinguishes it from arbitrary selection.

Thats the problem; we go on to talk about open, non-anonymous peer review as a particular solution, and about free software's methods of peer review and its value as an ideal for the practice of computer science at large. In the second post, I want to talk a bit about how badly, it seems to me, peer review is busted in the sciences. This will be anecdotal, insofar as I will be reliant upon my own experiences and observations. Still, considered as a report from the trenches, it might have some value for the reader. I should also qualify my comments by saying that while peer review seems to work reasonably well in journal article review, it is undeniably broke in conference article and grant proposal review, two fairly large and important parts of the practice of science today. We can then return to the solutions mentioned above.

Tale of a button

Ok, so this is a frivolous post (you weren't expecting a very serious one after this prolonged hiatus from blogging, were you?) I wear a "Free Software - It's all about freedom" button on my winter jacket (I picked it up at a swag dispensing sessions after a RMS talk a year or so ago). Here is a quick laundry list of the reactions I've gotten to this button (very few, and hence I can list them all):

#1. Student of mine sees me in the cafeteria, walks up, and says, "Free software? Where? I want some!". Interesting reaction; clearly, a free lunch is still attractive, but she seemed to be remarkably undiscriminating in her desire for software. Any software, so long as it was free, would do. I tried explaining the button, but I'm not sure I got anywhere.

#2. I visit a bar in Brooklyn for a friend's birthday party. A young man also invited to the party sees the button, and putting on a "save-the-whales" voice, speaks: "Set the software free, set the software free". I grin back, and he continues, "Information just wants to be free, doesn't it"? I'm getting tagged as a crunchy hippie here, but its allright, as more people crowd into the bar and disrupt our 'conversation'.

#3. Its my friend's daughter's 10th birthday party. His brother-in-law spots my button (I think he works downtown in Manhattan) and guffaws loudly. "You're such a dork, dude. Why don't you wear a peace button or something?". My intended response is cut off as my SO asks me if I want a drink. I shuffle off in pursuit of greater pleasures.

#4 And to bring things full circle. I run into the student mentioned in item #1 above. It's been a few months, memories have faded. For she says again, "Free software! Where? I want some!". And I try to return to my as-usual-futile explanation.