Confusion over the meaning of replication is harming social science, argues Michael Clemens. There has been a profound evolution in methods and concepts, particularly with the rise of empirical social science, but our terminology has not yet caught up. The meaning of replication must be standardized so that researchers can easily distinguish between replication efforts and the evaluation of robustness.
In Milan Kundera’s The Unbearable Lightness of Being, Sabina and Franz are doomed lovers. Kundera traces the demise of their relationship to a disagreement about what words mean. Sabina and Franz never realize that they mean different things when they say simple words—like “woman” and “truth.”
In development economics, a series of small wars have recently erupted between research colleagues. They involve disagreements over “replication” studies—studies that revisit earlier empirical research but arrive at different results. Similar battles have played out across the economics discipline. Part of these conflicts arise from the fact that different people mean different things when they say the word replication. That can be fixed, and I propose a way to do it in a new research paper.
Replication lies in the bedrock of science. The pathological science underlying “cold fusion” was uncovered after one lab claimed to generate boundless energy in a cheap, low-temperature chemical reaction but other labs could not replicate it. Thus in economics, when others learn that a famous result could not be “replicated,” they wonder about the original researchers—thinking at best of carelessness or freak chance, and at worst of deliberate fraud.
But often, studies that are perceived as attempting to replicate others’ work, and failing, are doing nothing of the kind. Rather, they get different results because they are asking fundamentally different questions, with different methods or different data. Again and again, the original authors have protested that the critique of their work got different results by construction, not because anything was objectively incorrect about the original work. (See Berkeley’s Ted Miguel et al. here, Oxford’s Stefan Dercon et al. here, and Princeton’s Angus Deaton here, among many others. Chris Blattman at Columbia and Berk Özler at the World Bank have weighed in on some of these controversies.)
Some of these rancorous, years-long battles would simply evaporate if social science had a single, clear definition of what is and what is not a replication. Then it would be clear when a follow-up study can claim that it fails to replicate an earlier result. But neither economics nor social science in general has such a definition.
Source: Clemens (2015) The Meaning of Failed Replications: A Review and Proposal – Working Paper 399
In my paper I put forward a modest fix: an unambiguous definition of what a replication test is. The current literature is hopelessly confused—I document 41 different and conflicting definitions!—and the definition I propose is one way to solve the problem. I also classify numerous recent critique papers, arguing that about two thirds of them are not replications by my definition.
It’s a modest step. But research colleagues, like Kundera’s lovers, make more progress when they know what each other are saying.
Featured Image: Figure Ground Ambiguity, Order, and Random by Sam Felder (CC BY-SA)
Note: This article gives the views of the author, and not the position of the Impact of Social Science blog, nor of the London School of Economics. Please review our Comments Policy if you have any concerns on posting a comment below.
Michael Clemens is a senior fellow at the Center for Global Development where he leads the Migration and Development initiative. His current research focuses on the effects of international migration on people from and in developing countries, and on rigorous impact evaluation for aid projects. He also serves as CGD’s Research Manager, directing the Center’s engagement with the academic research community through peer-review for Center publications, research seminars and conferences, and academic fellowship positions. He is a Research Fellow of IZA, the Institute for the Study of Labor in Bonn, Germany, and an affiliate of the Financial Access Initiative at New York University.