Since its onset, COVID-19 has significantly accelerated and expanded scientific publishing. Drawing on research into open peer review in medical journals, Serge P.J.M. Horbach discusses what impact COVID-19 has had on the practice of peer review and what shifting assessment thresholds for academic research on COVID-19 might suggest about the future of peer review itself.

As new waves of the COVID-19 pandemic continue to strike, people worldwide are wondering how we will get back to what was once called ‘normal’ and what the lasting effects of a year of restrictions, lockdowns and uncertainty will be. This also holds for the science and scientific publishing community. Following a strong increase in the number of submitted manuscripts to both pre-print servers and traditional journal outlets, shifts in gender distributions and research priorities, we are left to wonder what the enduring effect on science will be.

Scientific publishing is one aspect of the science system that has most certainly been affected by the Covid crisis. Among other changes, the criteria used to assess submitted manuscripts seem to have shifted. This impacts science at the core of its self-regulating and quality assurance mechanisms. The question remains whether it has changed for the good.

Early in the pandemic, scholarly journals and their publishers created fast-track systems for COVID-19 related content, enabling quick review and avoiding unnecessary publication delay. Studies have demonstrated these efforts to be highly effective, with average time between submission and publication being reduced by nearly 50% in samples of medical journals. While these efforts are obviously laudable, they raise questions regarding the content and quality of the review process: are Covid-19 related papers equally scrutinized and held against the same quality standards as other manuscripts? What are the lasting effects of these changes for editorial peer review? And what implications does this have for research quality and (public) trust in science?

This impacts science at the core of its self-regulating and quality assurance mechanisms. The question remains whether it has changed for the good

In my own research I have evaluated qualitative changes in the review process of manuscripts submitted to scholarly journals brought about by the Covid pandemic. Using the open peer review model at two medical journals, I compared manuscripts submitted before the pandemic, with those submitted during the pandemic, both for COVID-19 related, and non-COVID-19 related content. As it turns out, review for Covid-19 related papers does not seem to be less thorough – despite it being much faster – but we do see a noticeable shift in the quality criteria used to assess these manuscripts.

In evaluating scholarly manuscripts dealing with content related to COVID-19, journal editors, peer reviewers, and readers alike seem to be somewhat milder in their opinions, using different quality criteria and being satisfied with potentially lower standards. In particular, they seem to ask for fewer additional experiments, are satisfied with smaller samples, and suggest different strategies to address too strong claims. Authors are for example not asked to remove, extend or improve poor quality data, but rather encouraged to merely state this as a limitation, or even conclusion of their study.

While this is obviously an effective strategy to get results published faster, it can have multiple implications for scientific quality. While an abundance of potential causes of science’s apparent ‘reproducibility crisis’ have been proposed, small sample sizes and a lack of study power have been among the most frequently mentioned. Hence, refraining from requests to gather additional data might contribute to the wider spread of the irreproducibility, or the failure of early results translating into effective clinical interventions.

journal peer review has shown itself to be more agile then many might have expected. This in itself should be considered one of the pandemic’s positive outcomes

In a broader sense, review criteria seem to have shifted from prioritising novelty towards a focus on clinical or societal relevance. Especially when research concerns these topics of direct relevance to society and clinical applications, soundness and robustness of study findings should be prioritised. Due to changes in publication strategies, most prominently the rise of pre-print platforms, discussions about science’s reliability have become widely voiced anyway. Extending these concerns to results published in top-ranked journals, is likely not helpful to science’s quest to secure public trust in its well-functioning and reliability. Some highly publicised scandals have done considerable harm in this respect already.

The question remains whether these changes will have lasting effects on science, and, to begin with, whether we would consider that desirable. Evidently, the fact that science, and peer review in particular, seems to be able to adjust to new circumstances quickly is laudable. Attracting criticism about its inertia for decades, journal peer review has shown itself to be more agile then many might have expected. This in itself should be considered one of the pandemic’s positive outcomes. It demonstrates that assessment procedures are neither self-evident, nor carved in stone. With collaborative and cooperative efforts, they can be changed. Interestingly though, many of the changes were only witnessed for manuscripts related to COVID-19. Hence the effect of any changes in reviewers´ or editors´ attitudes seems very local.

A lack of clear differences in requirements for quality between pre-pandemic manuscripts and recent manuscripts not dealing with COVID-19, suggests that changes in reviewers´ and editors´ practices are not likely to be attributable to general shifts in gatekeepers’ attitudes that might have emerged during or due to the global pandemic. Similar results were found earlier regarding the speed at which review is performed.

This suggests that changes to editorial peer review and science publishing caused by the pandemic need not be permanent. However, they can be if we want them to. And importantly, the observed changes were not coordinated by large organisations or big commercial publishers, but rather emerged as consequences of individual academics’ choices. There is therefore an excellent opportunity to discuss the desired future of scholarly communication. We need to ask ourselves the important questions: Which direction do we want the system to evolve in? What processes do we need to assure the quality of the published record? And who should be involved in this? If anything, the pandemic has shown that the system can change. Now let’s use that to our advantage.

This post draws on the author’s published article, No time for that now! Qualitative changes in manuscript peer review during the Covid-19 pandemic, published in Research Evaluation.

Note: This article gives the views of the authors, and not the position of the Impact of Social Science blog, nor of the London School of Economics. Please review our Comments Policy if you have any concerns on posting a comment below.

Image Credit: Gerd Altmann via Pixabay