Journals play an important role in signalling the quality of academic research. This quality is often linked to measures such as the journal impact factor. However, these measures often obscure the overall quality of research papers in a journal. In this post, Margit Osterloh and Bruno Frey argue that the overall quality and originality of published academic research can be improved by introducing randomness into the peer review process.
The present publication process in academia is characterised by a “tyranny of top journals”. There are many biases in the review process. Evaluations by referees tend to be inconsistent, and the outcome of the review process only predicts the future impact of a scholarly contribution to a low degree. Moreover, empirical research has documented that to evaluate an article published in a “good” journal to be a “good” article is wrong in about two-thirds to three-quarters of all cases, due to the heavily skewed distribution of citations, ie. certain articles accounting for the majority of citations in a journal. This is true for both short citation windows and five-year spans. Nevertheless, top publications continue to have a powerful influence on promotion and tenure decisions.
empirical research has documented that to evaluate an article published in a “good” journal to be a “good” article is wrong in about two-thirds to three-quarters of all cases
How to stop this obsession with top journal publications? We advance a radical proposal, namely focal random selection of articles: When reviewers agree on the excellent quality of a paper, it should be accepted. Papers perceived unanimously as valueless are rejected immediately. Papers that are evaluated differently by the referees should be selected randomly for publication. Why random selection of contributions to which the referees do not agree? This procedure reduces the “conservative bias”, i.e. the bias against unconventional ideas. Where there is uncertainty over the quality of a contribution, referees have little evidence to draw on in order to make accurate evaluations. However, unconventional ideas may well yield high returns in the future. Under these circumstances a randomised choice among the unorthodox contributions is advantageous.
A drawback to this method is that inevitably some poor papers might be accepted. But, if path-breaking papers are many times more valuable than the poor papers are valueless, then randomisation will lead to a net gain. In fact the potential benefits are likely to be large enough that the losses are of little relevance. These considerations are corroborated by statistical tests involving two types of error: type I errors (“reject errors”) implying that a correct hypothesis is rejected, and type 2 errors implying that a false hypothesis is accepted (“accept errors”). The former matters more than the latter. “Reject errors” stop promising new ideas, sometimes for a long time, while “accept errors” lead to a waste of money, but may be detected soon once published. This is the reason why it is more difficult to identify “reject errors” than “accept errors”. Through randomisation the risks of “reject errors” are diversified.
Focal randomisation of controversial papers not only diversifies the risk of rejecting fruitful ideas, but in addition has an incentivising effect. It encourages researchers to submit unorthodox ideas that would otherwise have a hard time being published. Acknowledging the element of randomness in the selection process also promotes humility and more sociable research practices, by limiting the ability of individuals to dominate a particular field of research.
In contrast, the present publication system leads to lock-in and Matthew effects (“success breeds success”) and strengthens bureaucratic research governance. Unfortunately, this arrangement has gained much influence because a majority of authors benefit from it. Two third to three quarters of papers are overvalued, due to research systems that evaluate the quality of a single paper according to the quality of a journal. Therefore, appealing to scholars individually is not sufficient to change the present practice of performance management. Instead, proposals are needed for changes at the institutional level.
The purposeful use of random mechanisms in academia is not new. It played a role in the 18th century at the University of Basel. Vacant professorial chairs were filled by lot from a list of three candidates. At that time the main purpose was to weaken old boys´ networks. Today the main purpose is to ensure diversity that is crucial for the progress of scholarly work. The “tyranny of the top five” is de-emphasised, and the signalling function among a diversity of journals is redistributed.
Focal randomisation of controversial papers not only diversifies the risk of rejecting fruitful ideas, but in addition has an incentivising effect. It encourages researchers to submit unorthodox ideas that would otherwise have a hard time being published.
Such a system may possess some disadvantages. First, random procedures do not differentiate between good and bad quality. This is the reason why they are preceded by a pre-selection based on quality. It is important to note that the better the pre-selection works, the less the quality of the remaining papers can be distinguished. In this case, the variance in quality is reduced. It becomes much harder to decide which is “the best” or the “second best” paper. Through focal randomisation, the seeming disadvantage becomes an advantage, since otherwise personal preferences and unintended randomness might be decisive.
Second, more articles of low quality could be submitted if scholars knew that random selection played a role. Alternatively, it could be the case that more unorthodox high-quality articles would also be submitted, because authors would feel more emboldened than with the present system.
Third, random decisions are considered by many people to be “irrational”. However, seemingly rational decisions are often marred by many biases. Moreover, with focal randomisation scholars remain in power. They decide which papers are published or rejected immediately, and which enter the randomisation process. Focal randomisation unburdens editors considerably from the problem of dealing with low inter-rater reliability and contradictory reviews. In contrast to the unintended randomness attributed to the peer review process, which is sometimes close to an unintended lottery, this suggestion applies randomness in a strictly controlled and rational way. These proposals would benefit from field experiments which would likely, after a number of years, provide a compelling case for fundamentally changing the way researchers carry out peer review.
This post draws on the authors’ co-authored article, How to avoid borrowed plumes in academia, published in Research Policy.
Note: This article gives the views of the author, and not the position of the LSE Impact Blog, nor of the London School of Economics. Please review our comments policy if you have any concerns on posting a comment below.
Featured Image Credit Uncertainty, Momentum, via Wikimedia Commons (CC BY-SA 3.0)
This article ignores the wide range of reasons that papers are rejected by one reviewer but not another. One may be that the reviewed paper may be highly derivative of earlier work that was not referred to or the overlap not properly described. The rejecting reviewer may know of this work and the other not. The paper may have made an error in assumptions, experimental method, data analysis, or calculation and only one of the reviewers notices this error. These are objective criteria for rejection and these claims should be evaluated on merits, not by tossing a coin. It is only the evaluation of the “significance” of the work that is highly subjective for which the argument of this paper has any merit, in my opinion, and that is largely an issue in the most prestigious journals, which are well known to do a relatively poor job in evaluating the technical quality of articles.
The idea that novel and creative ideas cannot find a place to be published is, in my experience, a myth. There are now journals that explicitly only judge papers by technical soundness. There are preprint sites that will post papers without review. The claim of this article that publishing flawed papers only wastes funds is also incorrect. Those who are not experts in a field are less likely to detect flaws that passed one referee and misinformation will propagate and waste the time of scientists, particularly young ones who are less experienced. Of course bad stuff gets published now, but the proposal would increase the fraction of bad stuff in the literature.