Martyn Hammersley‘s provocative text seeks to interrogate the complex relationship between research, policymaking and practice, against the background of the evidence-based practice movement. Addressing a series of probing questions, this book reflects on the challenge posed by the idea that social research can directly serve policymaking and practice. Jennifer Miller finds this a thought provoking read.
The Myth of Research-Based Policy and Practice. Martyn Hammersly. SAGE. March 2013.
In the wake of the tragic mass shooting in Newtown, calls for gun policy reform echoed throughout the US. Almost a year on, little has changed, and Americans remain deeply divided on gun control policy. One option that was off the table, though, was evidence-based policy. Since 1996, when Congress blocked the Center for Disease Control’s (CDC) National Center for Injury Prevention from using its budget to “advocate or promote gun control”, virtually no research on gun safety has received federal funding.
The political tactic of blocking the use of research extends beyond gun policy. Just this year, Congress blocked the National Science Foundation from funding political science research unless it contributes to national security or economic competitiveness. In 2012, the North Carolina General Assembly blocked the use of the most current sea-level rise projections in the state’s coastal policy. In contrast to such political tactics mandating the exclusion of evidence, there appears to be a strong rationale for mandating evidence-based policymaking.
In The Myth of Research-Based Policy & Practice, Martyn Hammersley, Professor of Educational and Social Research at The Open University, suggests another reason why evidence-based policy may not be an option for gun policy reform. Perhaps evidence-based policy is only a myth. Hammersley describes the very name evidence-based policy as “a slogan whose rhetorical effect is to discredit opposition.”
In the first four chapters, Hammersley argues that social science research is too far removed from its applications in policy and practice, that advocates of evidence-based approaches are in fact advocating only for specific types of evidence, and that even the natural sciences fail to live up to their positivist reputation.
Yet in some ways Hammersley may underestimate the extent to which advocates of evidence-based policy are employing a political tactic. This tactical orientation seems especially evident where advocacy extends to legislating the types of acceptable research, such as in the examples he cites of the Reading Excellence Act and the Elementary and Secondary Education Act (No Child Left Behind). Each of these acts mandated “scientifically-based research” and attempted to define such research in terms of experimental design, quantitative analysis, and random assignment.
Advocates for evidence-based policy have hailed scientifically-based research as pointing toward “what works.” Drawing on Karl Popper’s philosophy of science, however, Hammersley points out that in its most positive, empirical form, science can only falsify. Thus, if we arrive at evidence-based policy from this positivist origin, at best science could only tell us definitively what does not work.
It is conventional wisdom not to attribute to malice that which can adequately be explained by ignorance. Hammersley has wisely outlined the ways in which advocacy for evidence-based policy may be based on ignorance of the nature of research, policy making, or both. But I am not sure I have been persuaded to rule out, if not necessarily malice on the part of evidence-based policy advocates, at least deliberate political maneuvering to advance an ideological position.
In the US, it is no secret that many seek to reduce the size and scope of government. In creating legislative mandates for specific types of evidence, such as randomized controlled trials (RCTs), advocates for evidence-based policy create a barrier to action. No matter the severity of a problem such as the racial achievement gap, should we take no action unless that specific action is supported by an RCT? Hammersly provides considerable evidence that evidence-based policy is not what it claims to be, but might have gone further in providing examples of its impact.
In Chapters 8-11, Hammersley turns his attention to systematic reviews and their alternatives, including traditional narrative reviews and qualitative synthesis. His critique of systematic reviews is that they double down on positivism, relying on its purported rationality to select and privilege certain types of studies and to establish the framework in which the review will be conducted. Hammersly claims that qualitative synthesis has great potential, but he acknowledges that a host of methodological issues within qualitative research must be resolved to realize that potential.
One of the few gun policy changes that has followed the Newtown shootings has been President Obama’s renewed support for gun safety research. One of the first fruits of that support has been a review of existing gun research by the Institute of Medicine. It is hard to imagine a more politically charged setting for a review. Between the setting and the sparse available research, a systematic review was probably not feasible. Would more systematic structure help the review hold up in the coming political battles? How might qualitative synthesis have yielded a richer result?
In the end, I find myself describing evidence-based policy as Churchill described democracy – the worst option excepting all others. Although this book dispelled some of the mythology, when it comes to evidence-based policy, to borrow a phrase from The X-Files, “I want to believe.”
Jennifer Miller is an Assistant Teaching Professor at the University of Southern California’s Sol Price School of Public Policy. She received her doctorate in public policy from the University of North Carolina at Chapel Hill. Her research interests focus on the scientific workforce. She has also written about collaboration among universities, industry, and government in university research centres. Before pursuing her doctorate, she worked for IBM in human resources. Read more reviews by Jennifer.