Replicability is considered a hallmark of good scientific practice, an important post-publication quality check. But how many studies are chosen for replication? Frank Mueller-Langer, Benedikt Fecher, Dietmar Harhoff, Gert G. Wagner have examined the economics literature and find that only one in one thousand publications are replication studies. The introduction of mandatory data disclosure policies may help to increase […]
Need a new search?
If you didn't find what you were looking for, try a new search!
The overall incidence of published replication studies in economics is minuscule – greater incentives are required
Some scholars have claimed that replication – the independent repetition of an earlier study, answering the same study question, using the same or similar methods under the same or similar circumstances – is not possible in the humanities. The reasoning is that the humanities search for cultural meaning can yield multiple valid answers, and that research objects are people […]
The replication crisis is largely concerned with known problems, such as the lack of replication standards, non-availability of data, or p-hacking. One hitherto unknown problem is the potential for software companies’ changes to the algorithms used for calculations to cause discrepancies between two sets of reported results. Anastasia Ershova and Gerald Schneider encountered this very problem in the course […]
Looking to solve the replication crisis in psychology? Limitations of questionnaire methods must be considered.
Throughout its history, psychology has been faced with fundamental crises that all revolve around its disciplinary rigour. Current debates – led in Nature, Science and high-ranking psychology journals – are geared towards the frequent lack of replicability of many psychological findings. New research led by Jana Uher highlights methodological limitations of the widely used questionnaire methods. These limitations may […]
Countering the claim that failed replications merely reflect the underlying truth of the scientific process and are not a matter of concern, Andrew Gelman argues that actually, the process is largely broken and we are in danger of dismissing the value of replication efforts. Current practice, centered on publication thresholds, is not filtering out poorly designed and executed studies.
John Snow points me […]
A clear distinction is needed between replication tests and the evaluation of robustness in social science literature
Confusion over the meaning of replication is harming social science, argues Michael Clemens. There has been a profound evolution in methods and concepts, particularly with the rise of empirical social science, but our terminology has not yet caught up. The meaning of replication must be standardized so that researchers can easily distinguish between replication efforts and the evaluation of robustness.
In Milan Kundera’s The Unbearable […]
Replication is not about making or breaking careers: it is about providing an opportunity to move science forward.
Replication and closer scrutiny of published findings are generally welcome in the scientific community, but questions have been raised over how replication attempts are being reported. Whilst there are certainly arguments for more friendly and cooperative tones to scientific debate, Dorothy Bishop welcomes this next chapter in rigorous debate. Reputation and career prospects will, at the end of the day, […]
Replication of government research uncovers shaky evidence on relationship between school and degree performance.
Interested in the statistical analysis used to justify the Department of Education’s reforms, Ron Johnston, Kelvyn Jones, David Manley, Tony Hoare and Richard Harris requested the data related to school performance and degree results via a Freedom of Information request. One year later the dataset was finally made available and they were able to identify some substantial flaws in the government research including sample […]
The FIRST Act’s demand for relevance at the expense of replication puts the entire scientific enterprise at risk.
The United States’ controversial FIRST Act would have profound implications for how social science research is managed and its funding allocated. David Takeuchi argues that even if the act doesn’t pass, it is clear that politicians are demanding more of a say in federally funded research. While a push to ensure research remains relevant can be a good thing, scientists and […]
A critical blind spot in the impact agenda has been that impact is understood and defined solely in positive terms. In this post Gemma Derrick and Paul Benneworth introduce the concept of ‘Grimpact’, to describe instances where research negatively impacts society, and argue that the implicit optimism of research assessment has rendered researchers and science systems poorly equipped to […]
Artificial Intelligence (AI) has been applied to an increasing number of creative tasks from the composition of music, to painting and more recently the creation of academic texts. Reflecting on this development Harry Collins, considers how we might understand AI in the context of academic writing and warns that we should not confuse the work of algorithms with tacit […]
Self-plagiarism, or publishing substantially similar work twice, is frowned upon in academia as a way of gaining an unfair advantage in a competitive ‘publish or perish’ environment. However, the increasingly open and digital nature of academic publishing means that replication is now easier than ever before. In this post, Mark Israel explores the ethics of self-plagiarism and asks, when is it […]
The future for academic publishers lies in navigating research, not distributing it
The world of scholarly publishing is in upheaval. As the open science and open research movements rapidly gain momentum, the access restrictions and paywalls of many publishers put them at odds with growing parts of the research community. Mattias Björnmalm suggests there is one way for publishers to once again […]
The growing, high-stakes audit culture within the academy has brought about a different kind of publishing crisis
The spate of high-profile cases of fraudulent publications has revealed a widening replication, or outright deception, crisis in the social sciences. To Marc Spooner, researchers “cooking up” findings and the deliberate faking of science is a result of extreme pressures to publish, brought about by an increasingly pervasive audit culture within the academy.
By now most readers will have heard […]
It is advisor attitudes that are likely to shape students’ attitudes towards questionable research practices
In debates on the validity of academic research findings, focus has been drawn to so-called questionable research practices, commonly understood to encompass a laundry list of behaviours that can increase the likelihood of statistically significant (and so more publishable) results. Anand Krishna and Sebastian M. Peter report on research examining attitudes to questionable research practices among students who have […]
Recent media reports in Germany have brought renewed focus on predatory publishing practices and seen a notably increased use of the term “fake science”. But to what extent is this a worsening problem? Lambert Heller argues that predatory publishing has never really become a big thing, and that it became a thing at all is largely attributable to the simple […]
Much of the social and medical sciences depend on randomised control trials. But while this may be considered the foundational experimental method, a certain degree of bias inevitably arises in any trial; whether this is sample bias, selection bias, or measurement bias. This is important as the level of validity of a trial’s causal claims can be a matter […]
Unhelpful, caustic and slow: the academic community should rethink the way publications are reviewed
The current review system for many academic articles is flawed, hindering the publication of excellent, timely research. There is a lack of education for peer reviewers, either during PhD programmes or from journal publishers, and the lack of incentives to review compounds the problem. Thomas Wagenknecht offers up some solutions to the current system, including encouraging associate editors to […]
Last week the Impact Blog featured a post from Richard P. Phelps, in which he proposed that journals get rid of their requirement for a literature review. Arnaud Vaganay agrees with much of what Phelps said, literature reviews are erratic and self-serving, but suggests doing away with them altogether is likely to make science less efficient and less credible. […]
The literature review is a staple of the scholarly article. It allows authors to summarise previous work in the field and highlight what makes their own contribution an original or novel one. But when those previous studies are misrepresented by an author, or even dismissed altogether amid claims of a “paucity of research”, isn’t the knowledge base in fact […]