Outlining their recent research into the different interests and commitments of groups looking to reform and improve scientific peer review, Ludo Waltman, Wolfgang Kaltenbrunner, Stephen Pinfield, and Helen Buckley Woods identify four schools of thought on the subject. Discussing their different aims and objectives, they highlight commonalities between them and also key areas in which they diverge. They suggest that in understanding these positions, it opens space for the purposeful inclusion of more varied forms of peer review for research.

Although peer review is generally seen as a central feature of the scholarly publishing system, it still brings with it widely-recognised problems – bias, time delays, risk aversion, and so on. A large variety of initiatives aimed at improving the peer review system have been developed over recent years, focusing on many different aspects of the system. These initiatives differ not only in how they aim to improve the system, but also in what they consider to be the key problems that need to be addressed.

Recent work on peer review that we have undertaken in the Research on Research Institute (RoRI) suggests to us that the landscape is shaped by four ‘schools of thought’:

• Quality & Reproducibility school
• Democracy & Transparency school
• Equity & Inclusion school
• Efficiency & Incentives school

As we discuss in this preprint, each school has a different view on the key problems of the peer review system and the innovations that are necessary to address these problems. Our identification of ‘schools’ is loosely inspired by an approach taken by Fecher and Friesike in their work on open science, but the schools we identify do not have a direct connection with their categories. Neither are they necessarily mutually exclusive. In some cases, the schools may be compatible, although in others, there are tensions.

Fig.1 Four schools of thought, each offering a different perspective on the problems of the peer review system and the innovations needed to address these problems, including indicative examples of key concerns for each school

The Schools

The Quality & Reproducibility school focuses on the role of peer review in evaluating and improving the quality and reproducibility of research. This school is interested in innovations in peer review that improve the quality of review reports and of published research. Examples include reviewer training, use of checklists, addition of a statistical reviewer, revealing of reviewer identities, and blinding of author identities. Reproducibility of research is also seen as key, with peer review playing an important role in this. Developments such as registered reports, in which peer review of a research plan and in-principle acceptance take place before carrying out data collection and analysis, have been introduced as an approach to improve the quality and reproducibility of research. Another focal issue for the Quality & Reproducibility school is safeguarding research integrity and identifying scientific misconduct.

The Democracy & Transparency school focuses on making the evaluation of scientific research more democratic and transparent. In the traditional journal peer review system, the evaluation of a manuscript is carried out by one or two editors together with a few anonymous peer reviewers. In an approach that is seen as more democratic, participation in the evaluation of a scientific work is open to a broader group of people. Soundness-only peer review, practised by open-access mega-journals such as PLOS ONE and Scientific Reports, is one of the most influential innovations in peer review in the last two decades. It is hailed by its advocates as more democratic, allowing the importance of an output to be determined by “the community”, rather than by small numbers of gatekeepers prior to publication.

Other innovations originating from the Democracy & Transparency school involve publishing peer review reports, making reviewer identities public or both, practices adopted by an increasingly large number of journals. Platforms such as F1000Research bring together the ideas of democracy and transparency by offering “publish then filter” approaches, in which a manuscript is first published and then peer reviewed in an open way.

The Equity & Inclusion school focuses on making peer review processes more equitable and inclusive. An example of a development initiated by this school is the work started recently in the context of the Joint commitment for action on inclusion and diversity in publishing, an initiative led by the Royal Society of Chemistry and supported by almost 50 publishers. The recent decision by IOP Publishing to move all its journals from single-blind to double-blind peer review is noteworthy. According to IOP Publishing, “Double-anonymous peer review – where the reviewer and author identities are concealed – has the potential to reduce bias with respect to gender, race, country of origin or affiliation which should lead to a more equitable system”.

The Efficiency & Incentives school focuses on improving the efficiency of peer review processes and the incentives for peer reviewers. This school is concerned about the pressure on the peer review system, which makes it increasingly difficult to find peer reviewers, causing the publication of new scientific results to be slowed down. To reduce the pressure on the peer review system, a large variety of initiatives have been developed to make the system more efficient and to incentivize researchers to contribute to the system.

Portable peer review, allowing reviews to be transferred between journals, offers a way to increase the efficiency of peer review. This commonly occurs with single publisher portfolios, but may also happen across different publishers, with some individual journals, such as BMC Biology, now allowing authors to submit review reports from other journals. Journal-independent peer review can be seen as a next step in the above developments, with an example being Review Commons. Initiatives to incentivize researchers to perform peer review are also promoted by this school, typically by giving more visibility and recognition to peer review activities, for instance using Publons and more recently, ORCID.

Complementarities and tensions between the schools

In some ways, the four schools of thought are complementary to each other. One may feel a strong commitment to the ideas of a particular school, but this does not mean that one cannot also be supportive of the ideas of other schools. For instance, someone’s primary focus may be on the need to improve the quality of peer review, but at the same time this person may also believe that peer review needs to be made more democratic and more equitable, and this person may also acknowledge that improving the quality of peer review will not be possible without making peer review more efficient. It is often a matter of emphasis rather than absolutes.

However, some of the ideas of the four schools are not compatible. Tensions between the various schools of thought pose a challenge because they may lead to conflicting views on how the peer review system can best be improved.

The identification of peer reviewers is an important example. The Democracy & Transparency school favours openness about reviewer identity and sharing of their reports as a way of creating accountability. The Equity & Inclusion school tends to favour double-blind peer review as a way of eliminating bias. These two seem incompatible. Moreover, the consequences of promoting double-blind peer review are that other developments, such as preprinting, are precluded; conducting genuinely anonymous peer review is impossible if versions of papers are already in circulation. Members of the Quality & Reproducibility school may also be sceptical of open approaches since reviewers might be less candid and less critical, which may harm research quality. They might also argue that reviewers should know the identity of authors in order to make properly informed judgements about the quality of a work, therefore advocating single-blind peer review. At the same time, members of the Efficiency & Incentives school may argue that openness in peer review makes it even more difficult to recruit reviewers, putting additional pressures on an already creaking system.

Tensions between the various schools of thought pose a challenge because they may lead to conflicting views on how the peer review system can best be improved.

Similar tensions can be seen to exist elsewhere. For example, the Quality & Reproducibility school favours a drive to create more robust peer review, and this may often add new dimensions to the review process, such as additional reviewers or specialist statistical review. This is obviously at odds with the Efficiency & Incentives school in its attempts to streamline review processes. The Efficiency and Incentives school may also be sceptical of moves favoured by the Equity & Inclusion school to increase the diversity of reviewer pools and editorial boards, as imposing targets on the diversity of peer reviewers may make it even more challenging to find reviewers, and diversifying editorial boards may disrupt the gift economy of peer review.

Is reconciliation possible?

There is a need, we believe, for more conversations between the different schools about the future development of the peer review system, and we hope our work might contribute to that. Such conversations may help find creative ways to deal with the tensions between the schools. At the same time, there is room for more heterogeneity in the peer review system, moving away from one-size-fits-all solutions, and allowing for the coexistence of different forms of peer review.

From a bigger picture view point, the connectedness of the peer review system with broader developments in the research system (e.g., the push for open science and responsible research assessment) highlights that improving peer review requires coordinated action by a multitude of stakeholders, not only scientific publishers, scholarly societies, journal editors, and meta-researchers like ourselves, but also funding agencies, research institutions, governmental organizations, and others. We hope that our work in the Research on Research Institute (RoRI) will help these stakeholders intensify their efforts to develop a rigorous evidence-informed understanding of the peer review system, to experiment with new forms of peer review, and to introduce improved peer review practices and policies.

The content generated on this blog is for information purposes only. This Article gives the views and opinions of the authors and does not reflect the views and opinions of the Impact of Social Science blog (the blog), nor of the London School of Economics and Political Science. Please review our comments policy if you have any concerns on posting a comment below.

Image Credit: Featured image, Geralt via Pixabay.