In a response to Jason Priem’s post advocating the use of Twitter by academics, Don Taylor writes that while Twitter, blogs and other social media should be part of academic life, we must not lose the slow, deliberative process that emphasizes thoughtful scholarship behind traditional publication in journals.
A few weeks back, the LSE Impact of Social Sciences blog published a piece on the use of twitter by academics, written by Jason Priem, that suggested peer review journals might become a thing of the past. Austin Frakt and I wrote a brief post noting that as much as we love the microblog service, the role of peer review journals cannot be replaced by Twitter, blogs, or anything else (and we really believe in blogs!). We need the slow deliberative process that emphasizes trying to get it right, as opposed to doing it quickly. We concluded:
We absolutely need the slow, peer review system as the foundation of thoughtful, careful scholarship. Twitter and other social media are important additions that can give scholarly content “reach” and “relevancy”. However, it’s a both/and, not an either/or proposition. Traditional peer review journals should remain the bedrock of the research evidence that can be brought to bear on health policy.
However, I think the peer review process often is too slow and could be sped up without losing precision. In addition, I think there is too much secrecy in the process and a bit more disclosure would likely be good (though there are likely plusses and minuses).
Following are a few personal thoughts about changes I would like to see in the peer review process used by journals that are based on my personal experience and preferences (I have published 70 peer review papers and reviewed dozens of manuscripts for journals). Others will likely have different thoughts, and I would be interested to know them. This is not meant to be a definitive word, just my personal thoughts.
- The identity of reviewer and reviewee should be known to one another
- The title of manuscripts under review should be public, along with the authors of the manuscript and the identity of the reviewers
- How long the reviewers have been reviewing the manuscript should be public
- How long authors have had a request for revision should be public
- Upon publication, the correspondence between reviewers/editors/authors should be public (this is important because often people say “why didn’t you do this subanalysis”; often it was done, but cut from a published paper due to length restrictions)
- The use of online early publication is a good thing; I wonder if it will eventually become the only modality? (I only take one journal in hard copy now, Health Affairs, and otherwise utilize Duke University’s global subscription service)
- Gated papers hinder academic investigation and discourse, but I am unsure of how to fund journals without subscriptions
Making the identify of all parties public and how long they have had to review a manuscript or complete revisions should provide some “speed” to process. More information about the give and take leading up to the publication would provide a fuller context for the paper. And a big issue going forward is the financial model by which journals survive, not to mention the question of who should pay for them and how much?
The problem I have with this is that you have conflated “peer-review” with “traditional peer-review journals”. The key objection to the current publication model, as I understand it, is that the actual review work done by editors and reviewers comes at no cost to the publisher – who in returns pays a relatively small amount for administrating the process, and then charges an ungodly amount for access. The actual value that the publisher adds has been dramatically reduced by the modern web – but instead of adapting to this changing environment, most publishers seem to be doing anything and everything to maintain their dominance – even if it damages science. It is now truly a case of the tail wagging the dog.
People who are calling for the end of “traditional peer-review publications” are not objecting to a process that is “deliberative”, “thoughtful”, or “careful”. They are simply objecting to a system whereby the publisher can act to maximize their own financial gain by promoting the merits of a process they make very little contribution to. These critics of traditional peer-review publications seek not only to maintain the current standards of peer-review, but to dramatically improve upon them, using new technologies and freedoms largely facilitated by the internet.
Lastly, too many assume that peer-review, as instituted by most publications, is actually any good. In my own experience, it’s terrible. I would guess that only about 5% of the reviews I see (as an author, or reviewer) are actually useful. I agree with most of your above points to improve peer-review, but they don’t address the key reason for bad peer-review – which is the massive contemporary increase in the number of papers needing to be reviewed by a shrinking body of people that have few incentives to prioritize those efforts over other actives such at teaching or research.
Darren’s comments are right on. If anything, he didn’t go far enough in criticizing the greed of journal publishers. For example, some publishers now want submitting authors to pay a fee amounting to hundreds, or even thousands (no exaggeration) of dollars so that their article will be “open access.” This is simply rampant greed, in my opinion. Consider the fact that journal contributors provide journals with free content. Unlike most periodicals, journals do not pay contributors for all their hard work, which in this case typically amounts to hundreds of hours of work by one or more researchers and writers working collaboratively. And while I’m not arguing that we should be paid for the articles we produce, I would argue that we shouldn’t be charged exorbitant fees to bring our work to the public’s attention either. Enough is enough.
Whether or not we need peer review might be a purely theoretical discussion. The slow pace of research–largely due to the publication stage–is creating a market for fast research that does not include peer review. Major grants are now going to organizations that can quickly implement interventions without IRB and publish results in lay reports without ever having undergone review. Speed itself is now a commodity and it is one antithetical to the structure of academic research.
One option I see for speeding up the research cycle is to merge peer review with research conferences. Paper reviewers could be selected and even paid in advance of the conference. The panel could meet in-person and come to consensus about the quality of research. Perennial conferences could gain Impact Factor that makes them more attractive for high-level researchers. The conference proceedings could publish within weeks instead of months, or worse, years.
When I raise this possibility, my academic colleagues often tell me peer review is the single most important aspect of the western scientific tradition. My argument is that academia is facing the same revolution from which journalism is just emerging. If we are unwilling to adapt peer-review, we will ensure its end. We are dinosaurs.