How to assess research impact? Sarah Morton draws on her own experience of assessing impact arguing that despite diverse topics, settings and countries, there are patterns of successful impact that can help move our learning forward. She outlines her five top lessons for designing and evaluating impact in a research project.
I am not alone in thinking that creating rich connections between researchers, practitioners, communities and policy-makers alongside a vision for how research can impact an area of work are the core elements of successful research impact. Since 2008 I have been involved in planning for impact, setting out impact pathways and assessing the impact of research on people, policy and practice in various roles at the University of Edinburgh, across national and international research initiatives and more recently through my own business, Matter of Focus.
In 2015 I published an empirically based framework for assessing research impact, and I have used it to assess impact in 7 studies including 14 case studies, of research centres and initiatives across 8 countries around the world. These include university-led, NGO and wider research initiatives. The most recent three have been international initiatives looking at children’s issues: impact of the Global Kids Online Programme run by the London School of Economics and UNICEF Innocenti, office of research; the Know Violence in Childhood global learning initiative, and the Multi-Country study of drivers of violence against children (UNICEF and University of Edinburgh).
Working across diverse structures and institutions in detailed case studies across 8 countries has led to deep insights into the business of planning for, creating and assessing the impact of social research. In this blog I pull together some of the insights from that work.
I want to pull these impact studies together for a number of reasons:
- There are some amazing successful impact examples that we can all learn from
- I want to close the loop between impact assessment and planning for impact – so that learning can help inform future approaches to Knowledge Exchange and Mobilisation (KE/KMb)
- The literature tells us that people and relationships are at the heart of impact – looking at impact studies can help see this in action
- Despite diverse topics, settings and countries, there are patterns of successful impact that can help move our learning forward.
What makes this set of studies interesting is the opportunity they give to draw lessons across a variety of contexts, topics, approaches and initiatives. The method is set out in my paper, but essentially uses a contribution analysis framework to set out and test a pathway to impact following this framework:
The Research Contribution Framework (Morton 2015)The most successful data collection methods involved a mix of documentary analysis, follow-up surveys and interviews with research users. Activity reports were important in setting out what had happened and who had been engaged. Where research users indicated changes in policy or practice, these were followed up in the relevant documents to identify evidence.
So what can we learn looking across these impact studies? I think there are five areas for learning:
1.Have a vision and intention for impact
All of the studies have been of research endeavours where impact was an important aim or value of the researchers and funders- perhaps unsurprisingly, people who commission impact studies believe that it is important that research has an impact on the people and communities whose problems and issues it investigates. Putting this intention front and centre and working hard to make it a reality are core features to successful impact. Where there was variable effort or focus on impact across a research portfolio, or where a funder had commissioned a post-hoc impact study rather than embedding it at the start it is much harder to find any evidence of impact, or indeed any clues for where impact may have happened.
2.Involve research users throughout
Relationships are at the heart of successful impact practice. The research impact literature shows that involving research users from the start increases the impact of research. This was really apparent in the impact studies, with research user engagement from early in projects or programmes clearly sharpening investigation, warming up potential wider stakeholder groups to engage with findings when they are ready, and helping to hone key messages for different audiences. Joint publishing with academics and non-academics has also been important. Those engaged in the research cycle also benefit from capacity building to understand and work with research.
3.Take the time and make the effort
It takes time and effort to craft and execute a successful impact strategy. Building relationships with stakeholders means finding, contacting and nurturing trust. In some cases, this has been a tension with the pressure from funders or institutions for a traditional research timetable and outputs. Knowledge mobilisation strategies mean engaging thoughtfully, and sticking with a project even when things aren’t going to plan. This really needs dedication, resources and a willingness to be open and learn.
4.Be honest about failure
Nearly all of the impact studies were focussed on successful impact but there were somewhere there was little or no evidence. It can be helpful to track and illustrate where there hasn’t been impact, or where engagement strategies have failed. Lack of impact was linked to lack of impact strategy, or to the breakdown of relationships, due to job changes, organisational pressures or lack of capacity to engage from either side.
5.Collect data and feedback as you go
The biggest barrier to understanding impact across these case studies was a lack of data. This was usually because people hadn’t collected it when they were in the midst of the projects. It is so much harder to go back and find it later – something I often talk about as a detective job! A few simple feedback mechanisms can easily be systematically threaded through any knowledge mobilisation strategy and then can form the bedrock for building up the story of impact.
I hope this inspires you to keep doing the important job of connecting research with people, communities and society. Please get in touch if you want to find out more.
Note: This article gives the views of the author, and not the position of the Impact of Social Science blog, nor of the London School of Economics. Please review our Comments Policy if you have any concerns on posting a comment below.
For a lot of us, the kind of ‘impact’ measured in the UK REF and other exercises is rather unsatisfactory. My own field is generally anti-capitalist, pro-economic degrowth, and strongly environmentalist. Success in impactful research would be measured in terms of halted or scaled back land grabs, mining and fracking projects, or fossil fuel-fired power generation harmful to the planet. Supporting social and environmental defenders, alternative economies, protest, and activism. The UK government, and others, do not like this sort of work, since they actively try to quell dissent surrounding capital intensive projects and proposals [and particularly, protest], and I doubt an ‘impact case study’ would ever be submitted by a university listing the kind of work that I do supporting Indigenous groups resisting mining on their lands, others in the climate extinction movement, [or for that matter an anarchist scholar helping to organise anti-capitalist demonstrations]. I could be wrong, but to what extent is ‘impact’ a mainstream neoliberal concept, as shown above? In my own writing I prefer to use the term ‘engagement’ and to divide it into activism, advocacy, and consciousness-raising work. I have a whole draft book listing these sorts of activities. Thankyou.
Rickards et al’s report, which identifies 3 generation of impact research and 7 provocations, is relevant. https://cur.org.au/cms/wp-content/uploads/2020/07/rickards-et-al-2020-research-impact-as-ethos_final-report.pdf