Discussions around improving peer review often focus on openness as a mechanism to reduce bias. Drawing on a recent study of double and single anonymisation at the British Ecological Society, Charles Fox argues for the benefits of double anonymisation as a means to reduce bias and especially institutional status bias in peer review.

Readers can find more LSE Impact Blog posts on Peer Review and other posts from Peer Review Week 2023 via the links. 

There is a widespread perception among scholars that systemic biases influence the outcomes of peer review of scholarly manuscripts. In particular, there is concern and evidence that the gender,  nationality, location or reputation of authors influences how manuscripts are assessed by reviewers due to unconscious, or conscious, biases.

To reduce these biases and make peer review fairer, some journals have adopted double-anonymous peer review, peer review in which the reviewers and authors are both anonymous. However, single-anonymous review, in which reviewers are anonymous but authors are not, remains the most common type of review in most disciplines. This is partly traditional, but also reflects skepticism that the benefits of double-anonymous review outweigh its costs and limitations.

An experiment: Double-anonymous peer review at Functional Ecology

In September 2019, my colleagues (Jennifer Meyer and Emilie Aimeé) and I began a randomized controlled experiment to examine how anonymising author identities influences the peer review process at the journal Functional Ecology. We wanted to collect the data necessary to elevate the relative benefits (reduction in bias), costs and limitations of double-anonymous review. Our study is not the first to examine how anonymising authors affects the peer review process at a scholarly journal, but is larger than most previous studies (~3700 reviewed papers over three years) and we collected more comprehensive data on authors, reviewers and the review process than in previous studies.

Our design was to randomly assign papers to be reviewed either with the authors’ names provided to reviewers (single-anonymous review) or with the authors anonymised (double-anonymous review). For papers in the single-anonymous treatment we included author details in emails sent to prospective reviewers and included the authors’ cover page in the manuscript sent to those that agreed to review. Author details were not provided to reviewers in the double-anonymous treatment.

The most striking result from our study is that authors residing in wealthier countries received a substantial advantage from being identified to reviewers. Papers written by authors residing in wealthier countries (countries with a high human development index, HDI) received higher scores from reviewers, and were more likely to be invited for revision, when the authors’ identities were known to reviewers (solid circles in Fig.1); their papers received lower scores, and were less likely to be invited for revision, when the authors’ were anonymous. Authors from less wealthy countries did not benefit from being identified to reviewers, though they also paid no cost to being identified to reviewers – their papers were reviewed similarly whether they were identified to authors or not.

We see the same overall pattern when comparing authors residing in countries with higher versus lower English-proficiency; authors from high English-proficiency countries get an advantage to being identified to reviewers, but this advantage is not shared by authors from non-English countries (Fig.2).

Our results demonstrate a substantial positive bias in which reviewers favour authors from wealthier English-speaking countries. Our design does not test specific causes of this positive bias. However, studies in a variety of contexts demonstrate that reviewers give higher ratings or are more likely to approve of the work of scholars with prestigious reputations, or from prestigious institutions, irrespective of the actual quality of the work. We hypothesize that a similar form of bias, based on the home country of authors, explains our results.

We found no evidence that review treatment differentially affected the evaluation of papers submitted by men or women. Papers authored by women did just as well, or even slightly better, as papers authored by men, regardless of review treatment.

Concerns about double-anonymous review

One concern we had about implementing double-anonymous review at Functional Ecology was that reviewers would be less likely to agree to review if they could not know the identity of authors. Instead, we found that reviewers were more likely to agree to review when the authors were anonymous. This shortened the time it took editors to recruit reviewers and thus reduced the average time from submission to decision by about 3.5 days.

Another common concern is that it can be very difficult to anonymise authors. In some disciplines it is possible to identify authors based on aspects of the research. Also, as authors increasingly submit their manuscripts to preprint servers before or concurrent with journal submission, their identity is publicly associated with their research before peer review. In post-review surveys during our study about 60% of reviewers tell us that they either knew or suspected the identity of the anonymised authors, and these guesses were fully or partially correct 90% of the time. Reviewers tell us they identified authors based on various combinations of the research location, study organism, methods used, and manuscript referencing, though few indicate that they had seen a non-anonymised version of the manuscript at another journal or as a preprint. Yet, despite frequent failures of the anonymisation process, we observed a large influence of review treatment on the influence of author demographic traits on peer review that should not be ignored.

Recommendations for other journals

Survey data show that scholars widely believe that evaluation of research is biased by characteristics of the authors, and that authors commonly prefer double-anonymous review as the best model for reducing these biases. Given our evidence that anonymising author identities makes peer review more fair, I think it prudent for journals to consider anonymising authors during peer review. Our data suggest there will be few challenges to doing so. Anonymising authors makes reviewers easier to recruit and, though authors can sometimes de-anonymise authors, these occasional failures of anonymisation do not counteract the benefits of improved objectivity of peer review. It is therefore pleasing that following the results of this trial, the British Ecological Society, who sponsored this research, is transitioning all of its journals to mandatory double-anonymous peer review

Some journals offer optional anonymisation of author identities, in which authors choose whether they are identified to reviewers or not. This is an easy step between single- and double-anonymous review, but is unlikely to substantially reduce reviewer biases because authors from high income countries – those that our data show get significant advantages from being identified to reviewers – are less likely to choose to be anonymous. Journals should just make the switch to double-anonymous review to make the process more equitable.

Double-anonymous review and open science

Based on our work and the large diversity of studies of unconscious bias, I believe that scholarly publishing will be more fair and equitable if journals anonymise author identities during peer review. However, some argue that adoption of double-anonymous review will hinder innovations in open science. Others argue that all participants in peer review should be identified to each other, a model often called open-identities review, making reviewers more accountable for their comments to improve both the tone and quality of reviewer comments.

But double-anonymous peer review need not be incompatible with all aspects of open review. There is nothing inherent in double-anonymous review that prevents open (but anonymous) interaction between authors and reviewers (e.g., collaborative peer review), or that prevents publishing reviewer comments and author responses alongside an accepted manuscript (open reports or transparent peer review).

Double-anonymous review can even be consistent with open-identities review if participant identities are not revealed until completion of the review process. However, there’s little evidence that identifying reviewers improves the tone and quality of reviews, and de-anonymising reviewers, even after publication, is not very popular with reviewers. Journals that adopt open-identities review see a decline in the proportion of invitees agreeing to review and, when signing reviews is optional, few reviewers sign their reviews. Identifying reviewers publicly, or privately to authors, would likely deter people from reviewing or discourage reviewers from being critical to avoid retaliation.

The content generated on this blog is for information purposes only. This Article gives the views and opinions of the authors and does not reflect the views and opinions of the Impact of Social Science blog (the blog), nor of the London School of Economics and Political Science. Please review our comments policy if you have any concerns on posting a comment below.

Image Credit: Paul Zoetemeijer via Unsplash.