Open peer review is often discussed more in theory than practice. Drawing on evidence from a recent systematic review of open peer review studies, Tony Ross-Hellauer and Serge P.J.M. Horbach find many persistent questions around open peer review remain poorly examined and call for a more experimental approach to open peer review practices.

In an era of transformative shifts in scholarly communication and growing calls for transparency, the institution of peer review faces increased scrutiny. Amidst concerns over accountability, transparency, and the equitable recognition of scholarly contributions, open peer review, whereby the identity of peer reviewers and/or their reports are openly published has been presented and adopted as a solution. However, there continue to be significant gaps in the knowledge base surrounding its efficacy and meaning amongst researchers.

Open peer review has broadly developed alongside other open science practices. A 2017 systematic review carried out by Tony identified three key innovations and pillars of the system (open identities, open reports, and open participation) and assessed the limited evidence for each. The review highlighted that much of the evidence pertained to stakeholder attitudes and expectations rather than concrete outcomes.

much of the evidence pertained to stakeholder attitudes and expectations rather than concrete outcomes

Proponents of open identities argued that revealing reviewers’ identities could enhance transparency and accountability, potentially reducing conflicts of interest and improving civility during the review process. Critics feared that signed reviews might lead to less critical feedback to avoid offending colleagues. Advocates for open reports suggested that making reports publicly available could improve scrutiny and quality assurance (particularly when coupled with Open Identities), bolster journal credibility and assist junior scholars in navigating the peer review process.

Since this review, significant strides have been made in making open science principles a reality and have reshaped the discourse on open peer review. To address this we carried out a comprehensive update to the 2017 review, undertaking a scoping review of evidence from 2017 to 2022, conducted using the PRISMA framework that delved into 52 relevant studies exploring open identities and open reports in open peer review. Open participation was excluded from the study, since hardly any new evidence on this aspect was found.

The current state of open peer review

The scarcity of evidence surrounding open peer review practices stems from its limited implementation. Our review sought to gauge the extent of adoption, revealing a gradual uptake of various elements of open peer review, notably open identities and open reports, albeit with variations across disciplines. We found open peer review remains far from common. Only a small fraction of journals, between one and five percent, have adopted these practices, although this includes several prominent outlets.

The review included several studies that evaluated the procedural impact of open peer review practices. Encouragingly, findings suggest that neither open identities nor open reports significantly affect reviewers’ likelihood of accepting invitations or their review completion times. However, concerns persist, particularly for smaller academic communities.

Attitudes towards open reports tend to be notably more favourable than those towards open identities, a trend observed across various studies. The main concern related to open identities is the fear of repercussions and backbiting following critical reviews, which could lead to blunting criticism, ultimately negatively affecting the peer review process.

our review found an overall dearth of studies addressing the crucial question of how open identities and open reports impact manuscript quality control and improvement

Limited evidence suggests shifts in reviewer behaviour, such as a tendency for reviewers to sign positive reports, with male and senior researchers more inclined to reveal their identities. Interestingly, neither format appears to deter reviewers from accepting review invitations, although uptake varies depending on disciplinary and geographical contexts, as well as the communication approach (opt-in or opt-out). There is initial evidence suggesting that open identities and open reports might lead to more constructive review reports, but this evidence is thin.

Despite these insights, our review found an overall dearth of studies addressing the crucial question of how open identities and open reports impact manuscript quality control and improvement: the primary objectives of peer review. This gap underscores the need for further research to elucidate the efficacy and broader implications of open peer review practices across diverse scholarly domains.

Open questions and concerns

Our findings indicate few drawbacks to publishing reports, for which there also seems to be general support from the research community. However, our study indicates that much further study is needed into open identities, given concerns for backbiting and blunted criticisms, for which there is little evidence.

Our findings indicate few drawbacks to publishing reports, for which there also seems to be general support from the research community.

Scholars therefore remain highly sceptical of revealing the identities of peer reviewers. In addition, the evidence points to potential ill-effects upon power dynamics and levels of visibility depending on who is more willing to review under these models and/or who is willing or able to sign their reports. There needs to be much more experimentation, including the evaluation of long-term effects, before widespread implementation can be responsibly pursued.

This lack of experimentation is pervasive, much remains to be studied and evaluated. Even though several research communities seem to be more open to various open peer review models, the effects of implementing these on the diversity of the reviewer pool, the quality of the review process, and decisions on what eventually gets published in which journals, remain largely unknown.

The need for experimental publishing

Effectively addressing these questions requires resources, including access to the relevant data, a research community that is willing to engage in coordinated experimentation, and a group of researchers willing to carry out these experiments.

Building on the findings we report here, along with Lex Bouter we recently published a “Call to Action” emphasising the need for further research. There we identified three priority areas for investigation:

• How do variants of open peer review affect the content and quality of the review process?
• What are the implications of open peer review elements for reviewers, authors, and other stakeholders involved in the review process?
• How do models of open peer review apply to review of objects other than traditional journal articles?

Study protocols to address some of these were developed by a strong international team during the 2023 Triangle Science Scholarly Communication Institute. However, much remains to be done. We call on the research community to engage in this process, and would invite you to contact us if you might wish to collaborate or support efforts on this.

The content generated on this blog is for information purposes only. This Article gives the views and opinions of the authors and does not reflect the views and opinions of the Impact of Social Science blog (the blog), nor of the London School of Economics and Political Science. Please review our comments policy if you have any concerns on posting a comment below.

Image Credit: Adapted from Bubble chamber event. Photograph taken December 16, 1958. Bubble Chamber-315B via National Archives and Records Administration (Public Domain).