Drawing on a new report, Deelan Maru and David Halpern outline the extent of under-evidenced public policy interventions and suggest how collaborative international approaches to building evidence architecture for policymaking could bridge this gap.

Governments spend trillions. Yet very little of this expenditure, or the programs and practises it funds, is based on robust evidence. In our recent report, we sought not only to quantify this gap, but also to show how governments could address it. In particular, how they might collaborate to build and share the evidence they have. 

The core issue is not new. In a classic study of the policymaking process from the early 90’s, Greenway et al wrote: 

“In each case, politicians justified their actions very much along the lines of ‘we’ve considered the options, weighed up the costs and benefits of each, and the only way to proceed is as follows’. Our case studies show that this was never the way in which it worked.” (Greenway et al, 1992, p218; quoted in Halpern and Wood, 1996)

This can’t just be pinned on politicians. There has been a systematic failure over decades around the generation and translation of policy-relevant research.

Let us start with gaps in the generation of primary research and evaluation. Randomised Controlled Trials (RCTs) in the social sciences are far fewer than those in health (see Fig.1). While RCTs are not the only type of study capable of evaluating interventions, it presents a stark depiction of how far the social sciences have to go in generating better evidence and evaluation. 

Fig.1: Number of RCTs in Health vs Social Science (produced by the Campbell Collaboration)

This gap may be reflected in the significant disparity in funding between these areas. R&D spending by governments in areas outside of health and defence is incredibly low, often averaging less than 0.5% of expenditure in the area. The implied research gap across the U.S., UK, Australia and Canada, is c.$100 billion per annum. 

Alongside low spend, governments are not evaluating as often as they should or as robustly. A 2019 report found that only 8% of UK government spending on major projects – £35 billion of £432 billion – had a robust evaluation plan in place. 

Conducting more primary research is evidently a key concern that governments should act on immediately. Collating available evidence (i.e. secondary evidence) is equally important.  However, while the quantity of secondary evidence has grown, quality remains variable. As one senior evidence producer put it: 

“Most things don’t work….the amount where there is enough there to justify public investment is very low (“really tiny”)… [and it is] lost in the noise from the clearing houses.”

More important than the generation of good quality primary and secondary evidence, is ensuring that it is put into the hands of policymakers and presented in a way that is easy for them to quickly digest the most appropriate action. 

“If we create a study, even an RCT, and it has a clear impact and we know the population and it’s scalable, we assume we’ve done our job. In our experience, that’s not the case at all. Even a quality research study is just another bottle floating in a sea of other research studies.”

Of course, the presentation of the evidence to policymakers only affects the supply-side. Without sufficient demand for evidence, academics and policymakers will continue to be misaligned. 

So, how do we fix these gaps in primary and secondary research production, and encourage adoption by policymakers? 

It’s worth noting the significant progress that has already been made. For example, the Evaluation Task Force in the UK, and the emerging Australian Centre for Evaluation. The What Works Centres and Clearing Houses across the UK and U.S. have also made great strides to synthesise evidence within policy areas and present it in ways that are useful to policymakers. Perhaps the best example is the Education Endowment Foundation, which produces toolkits for practitioners (teachers) assessing the quality of evidence and associated cost for different interventions:

Education Endowment Foundation Teaching and Learning Toolkit

In addition to governments simply increasing national R&D spend, a smart play would be for countries to collaborate to improve the evidence architecture. There are overlapping common interests and questions (e.g. how best to screen cancer, the best way to teach a child to read and write, and how to reduce recidivism) that apply across borders. Of course, caution needs to be taken when translating results, but surely an intervention that has worked in one country is a better starting point than nothing? 

To increase the number of evaluations being conducted across countries, a shared international evaluation fund should be established. From this fund, evaluation ‘paratroopers’ could be deployed across the world to evaluate promising new interventions. 

Alongside the fund, efforts to standardise reporting and publication protocols can help unlock further collation and comparison of evidence. To synthesise the available evidence, you first need to understand what evidence is out there. Here, evidence gap maps can play an important role. 

Crucially, synthesis needs transforming. A promising avenue for this are living evidence reviews – systematic reviews that are continuously updated. A living evidence review for every area of social policy would allow governments to rapidly assess what works in an area and adapt public services based on the latest evidence. 

Living evidence reviews are gaining traction. The Economic and Social Research Council in the UK has just announced £11.5m to fund improvements to evidence synthesis using AI. Wellcome has also announced $45m to develop new data and tools for accelerating living evidence synthesis. 

A living evidence review for every area of social policy would allow governments to rapidly assess what works in an area and adapt public services based on the latest evidence.

Another consistent finding from policymakers around the world is that research often fails to answer the question they have. Understanding whether ‘scared straight’ programmes work can be useful. But, policymakers also want to know what the best alternative interventions are. To do this evidence synthesis needs to operate at a higher level. Meta-Living Evidence Reviews, collating all evidence within a specific area present a way to represent policy areas as a whole are feasible, but again they require significant investment. 

From a UK perspective, surely one of the most impactful ways we can make progress on the new government’s Five Missions, would be to assemble the evidence we have, and don’t have, for each Mission. Evidence gap maps and meta-living evidence reviews should be commissioned for each of them as soon as possible. 

Boosting evidence adoption is laborious and requires organisational change. In the short term we should enable more cross-national sharing of knowledge. Medicine is particularly good at this, with societies, journals and mechanisms to accelerate the diffusion of best practice. We have some examples in other areas of policy, such as the Evidence for Education Network or the Society for Evidence Based Policing, but we could be much more systematic in convening these networks and provide them with the proper funding to become self-sustaining. 

From a UK perspective, surely one of the most impactful ways we can make progress on the new government’s Five Missions, would be to assemble the evidence we have, and don’t have, for each Mission.

Finally, we can take steps to get better at understanding the impact of research, i.e. building the evidence base for research itself (sometimes known as ‘metascience’). Further research can help us understand, for example: the returns to changing research expenditure; where best to direct funding; and how best to use AI in research. It can also help advance diffusion of evidence, e.g. what is the right level of human-centred dissemination versus toolkits and dashboards. 

There are nascent efforts to improve the global evidence ecosystem, and funding such as that put forward by the ESRC, will start to create transformational change. More can always be done. Equity is important. We have started with a coalition of the willing, largely anglophone high-income countries. We need to ensure that others are brought into the fold. The UN SDG Synthesis Coalition is a starting point. Variance is our friend. The more countries that collaborate, the more we can identify novel interventions that could translate to other contexts. 

Our report, available here, sets out the arguments above in detail. It is gratifying within weeks of its publication, c.$70m has already been committed to funding living evidence reviews. These are important developments for the better use of evidence in government, and for research translation and impact more generally. We hope it is the starting point for significant change in embedding evidence-based policymaking within governments.  

The content generated on this blog is for information purposes only. This Article gives the views and opinions of the authors and does not reflect the views and opinions of the Impact of Social Science blog (the blog), nor of the London School of Economics and Political Science. Please review our comments policy if you have any concerns on posting a comment below.

Image Credit: fran_kie on Shutterstock.