Responding to a recent paper supporting the implementation of double-anonymised peer review criteria as a means of reducing bias in favour of high profile academics, Serge P.J.M. Horbach, Tony Ross-Hellauer and Ludo Waltman, suggest open peer review may be more effective at bringing to light and mitigating different biases in peer review.

Discussions about bias in science have a long history. They have been the rationale behind several innovations in peer review formats both in publishing and in funding contexts. In a recent study that attracted a lot of attention, Jürgen Huber and colleagues once again ignited the debate about how to best address biases in peer review. While they argue for double-anonymised review, we believe more open review formats are a better solution.

Huber et al. examine the impact of ‘status bias’ in journal peer review practices. Using a paper co-authored by a Nobel laureate in Economics and a relatively unknown early-career researcher, they show that disclosing the identity of the Nobel laureate to potential reviewers makes them significantly more likely to accept the review invitation and to provide a favourable review. The authors attribute this to status bias and conclude that their results speak in favour of double-anonymised peer review. While we admire the study’s rationale, scale, and rigorous execution, we disagree with the authors’ conclusions.

openness helps to expose human biases and to discuss them publicly. It also allows for innovations in scholarly communication models.

We do not see double-anonymised review as the progressive answer to biases in peer review. We argue for sunlight instead of shadows: open peer review, with published review reports and optional open reviewer identities. Such openness helps to expose human biases and to discuss them publicly. It also allows for innovations in scholarly communication models. This is essential to tackle biases at the root, instead of just trying to minimise their consequences.

Innovations in scholarly communication

Double-anonymised peer review has far-reaching implications for publishing models. It hinders the adoption of important open science practices, including fast dissemination through preprint servers, early sharing of protocols and data sets, and transparency about competing interests. As such, double-anonymised peer review impedes innovation in scholarly publishing. Crucial innovative developments such as preprinting, preprint peer review, the publish-review-curate model, and micro-publications are all incompatible with double-anonymised peer review.

We see these innovative publishing models as a broader answer to concerns over biases in publication decisions. Reviewers at most journals are currently still tasked with assessing both methodological soundness and novelty or significance of submitted articles. Using double-anonymised peer review does not seem to affect the ability of reviewers to identify methodological shortcomings, but does influence articles’ acceptance likelihood. This suggests that peer review suffers mainly from biases in the assessment of novelty or significance, rather than methodological soundness. Biases in peer review seem to lie primarily in journals’ preference to publish the most impactful articles. Several of the innovative publishing models aim to minimise the focus on the ‘glamour’ of findings. In this way, these models are likely to reduce the effect of biases.

Limited and contextual effectiveness of anonymization

Secondly, the ability of double-anonymised review to address biases in peer review remains questionable, as was noted in Hilda Bastian’s response to Huber et al. While some studies show it might work in specific contexts, others demonstrate that reviewers are often capable of identifying authors, especially in smaller disciplinary communities. Double-anonymised review has a long history, but biases remain, also in contexts where double-anonymised review is the standard. This goes both for journal peer review and for funding peer review contexts. But worse, essentially double-anonymised peer review tries to minimise the consequences of biases without addressing their causes.

Historically, diverse research fields have had their own traditions regarding anonymity in peer review. In particular, many fields in the social sciences and humanities (SSH) prefer a double-anonymised review format, and have been slower than other fields in adopting review and publishing innovations.

Promoting equity, diversity and inclusion in science is crucial, but double-anonymised peer review is not a solution

Although the evidence is inconclusive, we understand the idea that anonymity in peer review might offer protection, especially in small research communities. However, on the whole, we argue that SSH fields in fact stand to particularly benefit from the increased transparency offered by new models like publish-review-curate. In the publish-review-curate model, articles are first published and then reviewed, after which they can be updated, and ultimately they can get a recommendation or endorsement from an editor or some other curator.

Interpretation and argumentation play stronger roles in SSH fields as compared to the physical and biomedical sciences. As Marcel Knochelman has well argued, peer review in the humanities depends less on “abstract, objectified quality”, being “rather a question of consensus and agreement of reviewers or editors on a particular level of intelligibility.” Hence, at least in some SSH disciplines, review processes might be best characterised as discussion, an exchange of viewpoints or opinions, or negotiation. In these contexts, transparency, at least regarding the content of review reports and editorial decisions, would have the important benefit of surfacing these discussions. The publish-review-curate model offers this transparency and strengthens the collaborative role of peer review in improving argumentation and interpretation.

Promoting equity, diversity and inclusion in science is crucial, but double-anonymised peer review is not a solution. We instead argue for sunlight, not shadows: more collaborative and open research cultures offer the best path towards more equitable and inclusive science.

The content generated on this blog is for information purposes only. This Article gives the views and opinions of the authors and does not reflect the views and opinions of the Impact of Social Science blog (the blog), nor of the London School of Economics and Political Science. Please review our comments policy if you have any concerns on posting a comment below.

Image Credit: Davide Cantelli via Unsplash.