LSE - Small Logo
LSE - Small Logo

Margit Osterloh

Bruno S. Frey

March 9th, 2020

To ensure the quality of peer reviewed research introduce randomness

3 comments | 33 shares

Estimated reading time: 10 minutes

Margit Osterloh

Bruno S. Frey

March 9th, 2020

To ensure the quality of peer reviewed research introduce randomness

3 comments | 33 shares

Estimated reading time: 10 minutes

Journals play an important role in signalling the quality of academic research. This quality is often linked to measures such as the journal impact factor. However, these measures often obscure the overall quality of research papers in a journal. In this post, Margit Osterloh and Bruno Frey argue that the overall quality and originality of published academic research can be improved by introducing randomness into the peer review process.


The present publication process in academia is characterised by a “tyranny of top journals”.  There are many biases in the review process. Evaluations by referees tend to be inconsistent, and the outcome of the review process only predicts the future impact of a scholarly contribution to a low degree. Moreover, empirical research has documented that to evaluate an article published in a “good” journal to be a “good” article is wrong in about two-thirds to three-quarters of all cases, due to the heavily skewed distribution of citations, ie. certain articles accounting for the majority of citations in a journal. This is true for both short citation windows and five-year spans. Nevertheless, top publications continue to have a powerful influence on promotion and tenure decisions.

empirical research has documented that to evaluate an article published in a “good” journal to be a “good” article is wrong in about two-thirds to three-quarters of all cases

How to stop this obsession with top journal publications? We advance a radical proposal, namely focal random selection of articles: When reviewers agree on the excellent quality of a paper, it should be accepted. Papers perceived unanimously as valueless are rejected immediately. Papers that are evaluated differently by the referees should be selected randomly for publication. Why random selection of contributions to which the referees do not agree? This procedure reduces the “conservative bias”, i.e. the bias against unconventional ideas. Where there is uncertainty over the quality of a contribution, referees have little evidence to draw on in order to make accurate evaluations. However, unconventional ideas may well yield high returns in the future. Under these circumstances a randomised choice among the unorthodox contributions is advantageous.

A drawback to this method is that inevitably some poor papers might be accepted. But, if path-breaking papers are many times more valuable than the poor papers are valueless, then randomisation will lead to a net gain. In fact the potential benefits are likely to be large enough that the losses are of little relevance. These considerations are corroborated by statistical tests involving two types of error: type I errors (“reject errors”) implying that a correct hypothesis is rejected, and type 2 errors implying that a false hypothesis is accepted (“accept errors”). The former matters more than the latter. “Reject errors” stop promising new ideas, sometimes for a long time, while “accept errors” lead to a waste of money, but may be detected soon once published. This is the reason why it is more difficult to identify “reject errors” than “accept errors”. Through randomisation the risks of “reject errors” are diversified.

 

An example of the uncertainty principle related to the relational interpretation. The more that is known about the position of a particle, the less is known about the velocity, and vice versa

 

Focal randomisation of controversial papers not only diversifies the risk of rejecting fruitful ideas, but in addition has an incentivising effect. It encourages researchers to submit unorthodox ideas that would otherwise have a hard time being published. Acknowledging the element of randomness in the selection process also promotes humility and more sociable research practices, by limiting the ability of individuals to dominate a particular field of research.

In contrast, the present publication system leads to lock-in and Matthew effects (“success breeds success”) and strengthens bureaucratic research governance. Unfortunately, this arrangement has gained much influence because a majority of authors benefit from it. Two third to three quarters of papers are overvalued, due to research systems that evaluate the quality of a single paper according to the quality of a journal. Therefore, appealing to scholars individually is not sufficient to change the present practice of performance management. Instead, proposals are needed for changes at the institutional level.

The purposeful use of random mechanisms in academia is not new. It played a role in the 18th century at the University of Basel. Vacant professorial chairs were filled by lot from a list of three candidates. At that time the main purpose was to weaken old boys´ networks. Today the main purpose is to ensure diversity that is crucial for the progress of scholarly work. The “tyranny of the top five” is de-emphasised, and the signalling function among a diversity of journals is redistributed.

Focal randomisation of controversial papers not only diversifies the risk of rejecting fruitful ideas, but in addition has an incentivising effect. It encourages researchers to submit unorthodox ideas that would otherwise have a hard time being published.

Such a system may possess some disadvantages. First, random procedures do not differentiate between good and bad quality. This is the reason why they are preceded by a pre-selection based on quality. It is important to note that the better the pre-selection works, the less the quality of the remaining papers can be distinguished. In this case, the variance in quality is reduced. It becomes much harder to decide which is “the best” or the “second best” paper. Through focal randomisation, the seeming disadvantage becomes an advantage, since otherwise personal preferences and unintended randomness might be decisive.

Second, more articles of low quality could be submitted if scholars knew that random selection played a role. Alternatively, it could be the case that more unorthodox high-quality articles would also be submitted, because authors would feel more emboldened than with the present system.

Third, random decisions are considered by many people to be “irrational”. However, seemingly rational decisions are often marred by many biases. Moreover, with focal randomisation scholars remain in power. They decide which papers are published or rejected immediately, and which enter the randomisation process. Focal randomisation unburdens editors considerably from the problem of dealing with low inter-rater reliability and contradictory reviews. In contrast to the unintended randomness attributed to the peer review process, which is sometimes close to an unintended lottery, this suggestion applies randomness in a strictly controlled and rational way.  These proposals would benefit from field experiments which would likely, after a number of years, provide a compelling case for fundamentally changing the way researchers carry out peer review.

 


This post draws on the authors’ co-authored article, How to avoid borrowed plumes in academia, published in Research Policy.

Note: This article gives the views of the author, and not the position of the LSE Impact Blog, nor of the London School of Economics. Please review our comments policy if you have any concerns on posting a comment below.

Featured Image Credit Uncertainty, Momentum, via Wikimedia Commons (CC BY-SA 3.0)


Print Friendly, PDF & Email

About the author

Margit Osterloh

Margit Osterloh is Permanent Visiting Professor at the University of Basel and Research Director at CREMA (Center for Research in Economics, Management and the Arts), Zürich. She was a full professor at the University of Zürich, Switzerland and Warwick, UK. She received a honorary doctorate in economics. Margit Osterloh is the author of numerous articles in academic journals and has written and edited many books in the fields of Management, Corporate Governance, Migration, Gender Economics, and other areas.

Bruno S. Frey

Bruno S. Frey is Permanent Visiting Professor at the University of Basel and Research Director at CREMA (Center for Research in Economics, Management and the Arts), Zürich. He was a full professor at the University of Konstanz (Germany), University of Zürich (Switzerland), and University of Warwick (UK). He was also a Visiting Research Professor at the University of Chicago. He received five honorary doctorates in economics. Bruno S. Frey is the author of numerous articles in academic journals and has written more than twenty books in the fields of Public Choice, Economics of Arts, Economics of Happiness, Economics of Terrorism, and other areas.

Posted In: Peer review

3 Comments