Replicability is considered a hallmark of good scientific practice, an important post-publication quality check. But how many studies are chosen for replication? Frank Mueller-Langer, Benedikt Fecher, Dietmar Harhoff, Gert G. Wagner have examined the economics literature and find that only one in one thousand publications are replication studies. The introduction of mandatory data disclosure policies may help to increase the likelihood of replication.

Academia is facing a quality challenge: the global scientific output doubles every nine years, while the number of retractions and instances of misconduct is increasing. In this regard, replication studies can be seen as important post-publication quality checks, in addition to the established pre-publication peer review process. It is for this reason that replicability is considered a hallmark of good scientific practice. But what can be done in order to increase the number and the share of replicated publications?

In our recent research paper, published in Research Policy, we explore how often replication studies are published in empirical economics and what types of journal articles are replicated. We find that between 1974 and 2014 just 0.1% of publications in the top 50 economics journals were replication studies. We provide empirical support for the hypothesis that higher-impact articles and articles by authors from leading institutions are more likely to be replicated, whereas the replication probability is lower for articles that appeared in top-five economics journals. Our analysis also suggests that mandatory data disclosure policies may have a positive effect on the incidence of replication.

Scientific research plays an important role in the advancement of technologies and the fostering of economic growth. Hence, the production of thorough and reliable scientific results is crucial from a social welfare and science policy perspective. However, in times of increasing retractions and frequent instances of inadvertent errors, misconduct, or scientific fraud, scientific quality assurance mechanisms are subject to a high level of scrutiny. Issues regarding the replicability of scientific research have been reported in multiple scientific fields, most notably in psychology. A 2015 report by the Open Science Collaboration estimated the reproducibility of 100 studies in psychological science from three high-ranking psychology journals. Overall, only 36% of the replications yielded statistically significant effects compared to 97% of the original studies that had statistically significant results. However, similar issues have been reported from other fields. For example, Camerer and colleagues attempted to replicate 18 studies published in two top economics journals — the American Economic Review and the Quarterly Journal of Economics — between 2011 and 2014 and were able to find a significant effect in the same direction as proposed by the original research in only 11 of 18 replications (61%). Considering the potential impact that economic research has on society, for example in a field like evidence-based policymaking, there is a particular need to explore and understand the drivers of replication studies in economics in order to design favourable boundary conditions for replication practice.

We explore formal (i.e. published) replication studies in economics by examining which and how many published papers are selected for replication and what factors drive replication in these instances. To this extent, we use metadata about all articles published in the top 50 economics journals between 1974 and 2014. While there are also informal replication studies that are not published in scientific journals (especially replications conducted in teaching or published as working papers) and an increasing number of other forms of post-publication review (e.g. discussions on websites such as PubPeer), these are not covered within our approach.

Between 1974 and 2014, 0.1% of publications in the top 50 economics journals were replication studies. We find evidence that replication is a matter of impact: higher-impact articles and articles by authors from leading institutions are more likely to be replicated, whereas the replication probability is lower for articles that appeared in top-five economics journals. Our results also suggest that mandatory data disclosure policies may have a positive effect on the replication probability.

Based on our findings, we argue that replication efforts could be incentivised by reducing the cost of replication, for example by promoting data disclosure. Our results further suggest that the decision to conduct a replication study is partly driven by the replicator’s reputation considerations.

Arguably, the low number of replication studies being conducted could potentially increase if replication studies received more formal recognition (for instance, through publication in [high-impact] journals), specific funding (for instance, for the replication of articles with a high impact on public policy), or awards. Since replication is, at least partly, driven by reputational rewards, it may be a viable strategy to document and reward formal as well as informal replication practices.

This blog post is based on the authors’ article, “Replication studies in economics—How many and which papers are chosen for replication, and why?”, published in Research Policy (DOI: 10.1016/j.respol.2018.07.019).

Featured image credit: Xiang Gao, via Unsplash (licensed under a CC0 1.0 license).

Note: This article gives the views of the authors, and may not in any circumstances be regarded as stating an official position of the European Commission, the LSE Impact Blog, nor of the London School of Economics.

About the authors

Frank Mueller-Langer is Senior Research Fellow at the European Commission, Joint Research Centre, Digital Economy Unit, Spain, and Affiliated Researcher at the Max Planck Institute for Innovation and Competition, Germany.

Benedikt Fecher is Head of the Research Project “Knowledge Dimension” at the Alexander von Humboldt Institute for Internet and Society, Germany, and Research Fellow at the German Institute for Economic Research (DIW Berlin), Germany.

Dietmar Harhoff is Director at the Max Planck Institute for Innovation and Competition, Germany.

Gert G. Wagner is Research Associate at the Alexander von Humboldt Institute for Internet and Society, Germany, Senior Research Fellow at the German Institute for Economic Research (DIW Berlin), Germany, and Max Planck Fellow at the MPI for Human Development, Germany. His ORCID is: 0000-0002-5985-4073.

–