Over the last twenty or so years, it has become standard to require policy makers to base their recommendations on evidence. That is now uncontroversial to the point of triviality – of course, policy should be based on the facts. But are the methods that policy makers rely on to gather and analyse evidence the right ones? Evidence-Based Policy contends that the dominant methods which are in use now – methods that imitate standard practices in medicine like randomised control trials – do not work. Michael Bassey believes policymakers should engage in the kind of critical and analytical processes advocated by this book before rolling out social changes.
Evidence-Based Policy: A Practical Guide to Doing it Better. Nancy Cartwright and Jeremy Hardie. Oxford University Press. 2012.
In March this year the UK government announced the launch of a network of six “What Works” centres aimed at providing robust evidence to inform policy making. The rationale was given as “It is a fundamental principle of good public services that decisions are made on the basis of strong evidence and what we know works. Yet all too often evidence is not presented in a simple, relevant format that enables it to be used to its maximum potential by service providers, commissioners and policymakers.”
This fascinating book was published a few months earlier, although it was unmentioned in the government announcement. It is a tough read, but it should be the standard text for these centres and on the reading list of all nineteen British university courses in PPE (Philosophy, Politics and Economics) – i.e. the breeding ground of politicians: but too late for David Cameron and Ed Miliband who both read PPE at Oxford. In the authors’ own words this is why it deserves attention:
“You are told: use policies that work. And you are told: RCTs – randomized controlled trials – will show you what these are. That’s not so. RCTs are great, but they do not do that for you. They cannot alone support the expectation that a policy will work for you. What they tell you is true – that this policy produced that result there. But they do not tell you why that is relevant to what you need to bet on getting the result you want here. For that, you will need to know a lot more. That’s what this book is about. We are going to show what else you have to have and how you set about finding it.”
In essence, Nancy Cartwright and Jeremy Hardie discuss how to get from “it worked there” to “it will work here”. This they describe as the “effectiveness” of a policy and is the sole focus of the book. They recognise that it is just one of the many factors, albeit a vital one, that determine policy decisions. Others include costs and benefits, available resources, alternative choices, ideology, and political expediency.
In examining whether it will work “here”, Cartwright and Hardie argue that one needs first to find the causal principles that link “there” with “here”. Then search for the support factors that played a positive causal role in making “it” happen “there”. The authors call this the “horizontal search”, which leads to asking whether these factors are present “here”. Next is the “vertical search” for the level of abstraction that will link “there” and “here” and so identify what will genuinely play a positive causal role “here”. Yes, as stated earlier, this book is a tough read and while I fear that my précis does thin justice to the process described in the book, my judgement is that it is exactly the kind of critical and analytical process that policymakers should engage in before rolling out social changes.
To help their argument Cartwright and Hardie use some powerful examples. Starting in 1985 the STAR project in Tennessee showed, as demonstrated by a randomized controlled trial, that students in the experiment’s smaller classes performed better at K-3 grade levels than did students in the larger classes. It also showed that minority and inner-city children gained two or three times as much from reduced class sizes as did their white and non-urban peers. Ten years later the state of California had problems with its early school grades finding itself at the bottom of the 39 states in the 1994 National Assessment of Education Progress. Reducing class sizes fitted with popular opinion, with common sense and the Tennessee RCT gave crucial evidence that it worked. In hindsight, of course, it worked “there”. California spent $1 billion, rising to $1.6 billion, on establishing half-size classes throughout the state within a year. But the positive results that were expected did not follow. Rigorous evaluation by 2002 found no conclusive link between reducing class size and the achievement of students. Moreover there was no improvement for disadvantaged children.
The Californian policymakers had not done an effective horizontal search. In Tennessee only schools that had available space to increase the number of classes were involved. In California, with so many schools involved, there was often insufficient spare space and so it was taken from other school activities – special needs, music and arts, athletics and child care programs. In Tennessee there was no shortage of qualified teachers to staff the reduced size classes, but in California an additional 12,000 teachers were hired quickly and many of these were unqualified. Moreover by limiting other school activities the policy had some negative effects on the school population.
Here is another example of unintended consequences. It may seem obvious that an isolated study space at home will improve a young person’s homework. But this presumes high motivation for homework. As Cartwright and Hardie point out, it may also be just what is needed to enable a badly motivated child to spend time texting her friends which she could not do if she were working in the same room as her family.
The UK Coalition government is introducing changes in the NHS, probation service, school education and elsewhere at breakneck speed and with little sign of evidence-based policy. Its “What Works” centres are being established too late. It’s a tragedy that this book wasn’t available when our current ministers were undergraduates. I recommend it to all who want to see policy improve social practice.
Michael Bassey is an emeritus professor of Nottingham Trent University and an academician of the Academy of Social Sciences. His methodological text Case Study Research in Educational Settings has sold over 5000 copies. Read more reviews by Michael.