Responding to a recent paper supporting the implementation of double-anonymised peer review criteria as a means of reducing bias in favour of high profile academics, Serge P.J.M. Horbach, Tony Ross-Hellauer and Ludo Waltman, suggest open peer review may be more effective at bringing to light and mitigating different biases in peer review.
Discussions about bias in science have a long history. They have been the rationale behind several innovations in peer review formats both in publishing and in funding contexts. In a recent study that attracted a lot of attention, Jürgen Huber and colleagues once again ignited the debate about how to best address biases in peer review. While they argue for double-anonymised review, we believe more open review formats are a better solution.
Huber et al. examine the impact of ‘status bias’ in journal peer review practices. Using a paper co-authored by a Nobel laureate in Economics and a relatively unknown early-career researcher, they show that disclosing the identity of the Nobel laureate to potential reviewers makes them significantly more likely to accept the review invitation and to provide a favourable review. The authors attribute this to status bias and conclude that their results speak in favour of double-anonymised peer review. While we admire the study’s rationale, scale, and rigorous execution, we disagree with the authors’ conclusions.
openness helps to expose human biases and to discuss them publicly. It also allows for innovations in scholarly communication models.
We do not see double-anonymised review as the progressive answer to biases in peer review. We argue for sunlight instead of shadows: open peer review, with published review reports and optional open reviewer identities. Such openness helps to expose human biases and to discuss them publicly. It also allows for innovations in scholarly communication models. This is essential to tackle biases at the root, instead of just trying to minimise their consequences.
Innovations in scholarly communication
Double-anonymised peer review has far-reaching implications for publishing models. It hinders the adoption of important open science practices, including fast dissemination through preprint servers, early sharing of protocols and data sets, and transparency about competing interests. As such, double-anonymised peer review impedes innovation in scholarly publishing. Crucial innovative developments such as preprinting, preprint peer review, the publish-review-curate model, and micro-publications are all incompatible with double-anonymised peer review.
We see these innovative publishing models as a broader answer to concerns over biases in publication decisions. Reviewers at most journals are currently still tasked with assessing both methodological soundness and novelty or significance of submitted articles. Using double-anonymised peer review does not seem to affect the ability of reviewers to identify methodological shortcomings, but does influence articles’ acceptance likelihood. This suggests that peer review suffers mainly from biases in the assessment of novelty or significance, rather than methodological soundness. Biases in peer review seem to lie primarily in journals’ preference to publish the most impactful articles. Several of the innovative publishing models aim to minimise the focus on the ‘glamour’ of findings. In this way, these models are likely to reduce the effect of biases.
Limited and contextual effectiveness of anonymization
Secondly, the ability of double-anonymised review to address biases in peer review remains questionable, as was noted in Hilda Bastian’s response to Huber et al. While some studies show it might work in specific contexts, others demonstrate that reviewers are often capable of identifying authors, especially in smaller disciplinary communities. Double-anonymised review has a long history, but biases remain, also in contexts where double-anonymised review is the standard. This goes both for journal peer review and for funding peer review contexts. But worse, essentially double-anonymised peer review tries to minimise the consequences of biases without addressing their causes.
Historically, diverse research fields have had their own traditions regarding anonymity in peer review. In particular, many fields in the social sciences and humanities (SSH) prefer a double-anonymised review format, and have been slower than other fields in adopting review and publishing innovations.
Promoting equity, diversity and inclusion in science is crucial, but double-anonymised peer review is not a solution
Although the evidence is inconclusive, we understand the idea that anonymity in peer review might offer protection, especially in small research communities. However, on the whole, we argue that SSH fields in fact stand to particularly benefit from the increased transparency offered by new models like publish-review-curate. In the publish-review-curate model, articles are first published and then reviewed, after which they can be updated, and ultimately they can get a recommendation or endorsement from an editor or some other curator.
Interpretation and argumentation play stronger roles in SSH fields as compared to the physical and biomedical sciences. As Marcel Knochelman has well argued, peer review in the humanities depends less on “abstract, objectified quality”, being “rather a question of consensus and agreement of reviewers or editors on a particular level of intelligibility.” Hence, at least in some SSH disciplines, review processes might be best characterised as discussion, an exchange of viewpoints or opinions, or negotiation. In these contexts, transparency, at least regarding the content of review reports and editorial decisions, would have the important benefit of surfacing these discussions. The publish-review-curate model offers this transparency and strengthens the collaborative role of peer review in improving argumentation and interpretation.
Promoting equity, diversity and inclusion in science is crucial, but double-anonymised peer review is not a solution. We instead argue for sunlight, not shadows: more collaborative and open research cultures offer the best path towards more equitable and inclusive science.
The content generated on this blog is for information purposes only. This Article gives the views and opinions of the authors and does not reflect the views and opinions of the Impact of Social Science blog (the blog), nor of the London School of Economics and Political Science. Please review our comments policy if you have any concerns on posting a comment below.
Image Credit: Davide Cantelli via Unsplash.
I have edited a social science journal now for 20 years and published 600 articles, plus rejected a lot of others. We have always had double blind review and it has worked fine. Were there a voluntary ability to reveal your name, I suspect only a tiny minority would avail themselves of it. One reviewer in the last five years has done this.
As editors we take out the snarky offensive comments from reviews, so that is not a problem. So people may think they are hiding behind anonymity, but their worst irrelevant remarks and “cite me in the paper!!!” comments are not passed on.
What we really need is a greater response rate from reviewers in the profession. I am growing tired of asking 15 people per article to do reviews. The sense of solidarity that non-anonymous reviewing tries to create, could usefully start with academics actually committing to do say 20-30 reviews a year in the first place each year. As a bit of a generalist, I do this myself across many journals and it is satisfying [my politics is that I will never review for Elsevier].
Thanks for your viewpoint – there are certainly disciplinary differences here and double-anon review in much broader use in social sciences. I’d also be interested to know, though, how do you know its “working fine” to avoid status biased? Have you analysed data or run experiments? Also, note open peer review means different things to different people and here we’re more interested in the transparency that comes with publishing review reports, with or without reviewer identities being revealed.
I was an editor for a decade on a top social science journal where I desk-rejected about half of the submissions on the basis of either quality or scope or both, and where I relied on three reviewers for each paper drawn in almost all cases from the reference lists (bibliographies) of the submitted manuscripts. Obviously I knew who the reviewers and the authors were (although I rarely knew them in person or even by repute because this particular journal area covered a very broad area and was thoroughly international in scope), but anonymity otherwise prevailed. As editor, my job was to ensure that the reviews were as fair and objective as possible, and giving due recognition to novelty even where other features of the submissions might be weaker.
The trouble with opening up name recognition is that you immediately introduce a dynamic of contextual awareness that is otherwise absent and that threatens to cut across the carefully controlled and moderated judgement process. I do not remember a single reviewer or author wishing to change this system, and I handled a new submission every day with 20-25 to process at weekends. The way to deal with the issues raised in this blog is to publish data: alongside time from submission to publication we could and should provide statistics on published papers and desk and post-review rejects broken down in key areas of concern such as gender, ethnicity and place of affiliation, controlling for other factors.