LSE - Small Logo
LSE - Small Logo

Laura Meagher

David Edwards

September 18th, 2020

How to tell an impact story? The building blocks you need

2 comments | 99 shares

Estimated reading time: 10 minutes

Laura Meagher

David Edwards

September 18th, 2020

How to tell an impact story? The building blocks you need

2 comments | 99 shares

Estimated reading time: 10 minutes

At a loss for how to demonstrate impact? Laura Meagher and David Edwards outline a dynamic understanding of impact evaluation comprised of ‘building blocks’. These building blocks are five types of impacts; five broad categories of stakeholders; and eight causal factors, along with a set of over-arching reflective questions.


The increasing pressure placed upon research organisations of all kinds (not just universities) to demonstrate impact is often passed along to researchers and research managers without the support of an accompanying cultural change within the institution.

With this in mind, Forest Research, the government research agency that supports the UK forestry sector, has committed to promote a culture of knowledge exchange and interdisciplinarity in its research. This mission stemmed from a 2012 external impact evaluation which recommended directing attention toward helping researchers generate and record impacts. In addition to programme activities such as training workshops and small-group conversations, the initiative identified an additional challenge: that of demystifying and operationalising impact evaluation in a coherent and accessible way that nonetheless allows for diversity, subtlety and uncertainty.

The initiative identified an additional challenge: that of demystifying and operationalising impact evaluation in a coherent and accessible way that nonetheless allows for diversity, subtlety and uncertainty.

A new evaluation framework: building blocks

To undertake this almost contradictory challenge, we developed a new evaluation framework that we tested across multiple projects in a pilot exercise.  Working with the draft framework made it possible to produce a dozen full impact case studies from initial observations. Analysis of this exercise indicated that the components of the framework were sufficiently clear and comprehensive for researchers and stakeholders to understand and use in constructing a range of very different narratives.

We envisioned the framework as a set of building blocks which could be combined in multiple ways to tell almost any impact story, each shedding light on the complex interactions between research, policy and practice. In fact, when we gave a demo at the 2019 UK Knowledge Mobilisation Forum, we represented each component with different wooden tiles (fitting for work with a forest research agency!) and then used them to tell two contrasting impact case study narratives, showing how we could manipulate the tiles into different configurations to reveal the relationships between them. The audience seemed to appreciate our main messages of dynamics – in terms of multi-directional interactions unfolding in varying sequences – and heterogeneity – in terms of impact types, causes and stakeholders   – as an alternative or complement to conventional linear logic models that tend to dominate the field of impact evaluation.

Impact Evaluation Framework ‘Building Blocks’

What are the building blocks?

There are: five types of impacts; five broad categories of stakeholders; and eight causal factors, along with a set of over-arching questions.

Five impact types

From previous impact evaluations and other research, we had found that providing a range of impact types to consider often makes people feel ‘liberated’ and ‘authorised’ to go beyond conventionally-sought instrumental changes when searching for their impacts. The five impact types are:

1) Instrumental: changes to plans, decisions, behaviours, practices, actions, policies

2) Conceptual: changes to knowledge, awareness, attitudes, emotions

3) Capacity-building: changes to skills and expertise

4) Enduring connectivity: changes to the number and quality of relationships and

trust

5) Culture/attitudes towards knowledge exchange, and research impact itself

There is no fixed hierarchy or linear sequence inherent in this typology, and all have value in their own right, although the last two, process-based impacts may make other sorts of impacts more likely.

Asking ‘How do we know?’ leads towards gathering of evidence, including recognition of telling indicators of change.

We envisioned the framework as a set of building blocks which could be combined in multiple ways to tell almost any impact story, each shedding light on the complex interactions between research, policy and practice

 

Five categories of stakeholders

To stimulate further definition (or mapping) for a particular situation, the framework offers five broad categories of stakeholders, which of course can be subdivided further, and may influence as well as be influenced by the research process and its outputs.

1)Policy-makers: government agencies and regulatory bodies, local, national and international

2)Practitioners: public, private, NGOs

3)Communities: of places or interest, general public

4)Researchers: within and beyond the project and the institution

5)Other

 

Eight causal factors

Because impact generation is a dynamic process, it is important to ask: ‘Why/how did changes occur?’. The framework offers eight causal factors:

1) Problem-framing: Level of importance; tractability of the problem; active negotiation of research questions; appropriateness of research design.

2) Research management: Research culture; integration between disciplines and teams; promotion of research services; planning; strategy.

3) Inputs: Funding; staff capacity and turnover; legacy of previous work; access to equipment and resources.

4) Outputs: Quality and usefulness of content; appropriate format.

5) Dissemination: Targeted and efficient delivery of outputs to users and other audiences.

6) Engagement: Level and quality of interaction with users and other stakeholders; co-production of knowledge; collaboration during design, dissemination and uptake of outputs.

7) Users: Influence of knowledge intermediaries, e.g. ‘champions’ and user groups; incentives and reinforcement to encourage uptake.

8) Context: Societal, political, economic, biophysical, climate and geographical factors.

 

Reflective questions

The questions underlying the framework are: ‘Who or what changed? (And how do we know?); Why or how did changes occur? and What lessons can be learned?’ 

These are centred around two questions which encourage the framework’s users to step back and think about what lessons have been learned regarding impact identification and generation, to plan actions and improve future efforts:

1) What worked? What could (or should) have been done differently?

2) What could (or should) be done in the future?

The framework can be used to stimulate critical reflection by individual researchers or teams, ideally involving end-users and other stakeholders, throughout the project cycle, at the planning, mid-course and final stages.

We tried to encourage researchers to see impact as complex, developing over often lengthy periods of time, in a dynamic context in which different causal factors, stakeholders and impacts themselves interact.

From simplicity to complexity

When assessing impacts, there can be a temptation to package notions of impact into readily countable and comparable units which come about as a result of a simple linear sequence of events. Instead, we tried to encourage researchers to see impact as complex, developing over often lengthy periods of time, in a dynamic context in which different causal factors, stakeholders and impacts themselves interact. Ultimately, impacts are highly heterogenous and travel different developmental pathways. Their stories are multi-dimensional and based on complex interactions.

Forestry was an ideal field in which to develop a framework of this kind, as it draws together a ‘real world’ orientation with an interdisciplinary approach spanning the natural and social sciences. However, we see the framework as having a wider application and being flexible enough to meet the needs of other disciplines and research organisations.

The ‘building blocks’ (impact types, stakeholders and causal factors) we identified can be assembled in multiple combinations to construct nuanced impact narratives in a way that is helpful for accountability (e.g. for funders), communication (e.g. for stakeholders) and learning (e.g. for researchers/managers).

Whatever their organisation, sector or country, we encourage readers to try it out and judge for themselves!

 


This post draws on the authors’ co-authored paper:

David M. Edwards and Laura R. Meagher. 2019. Forest Policy and Economics. A framework to evaluate the impacts of research on policy and practice: A forestry pilot study. Forest Policy and Economics. https://doi.org/10.1016/j.forpol.2019.101975


Note: This article gives the views of the author, and not the position of the LSE Impact Blog, nor of the London School of Economics. Please review our comments policy if you have any concerns on posting a comment below.

Image credit: Author’s own 

Print Friendly, PDF & Email

About the author

Laura Meagher

Laura Meagher PhD is the Senior Partner in the Technology Development Group, an Honorary Fellow at the University of Edinburgh and at the James Hutton Institute and an Associate at the Research Unit for Research Utilisation at the University of St Andrews. She has spent over 30 years working in the US and the UK with and within research and education institutions, along with industry and government, focussing on facilitation and evaluation of strategic change. Having herself led or facilitated multiple knowledge exchange initiatives, she has for many years evaluated impacts of research in diverse fields and sectors. Increasingly, her work incorporates an international dimension. She has also led, facilitated, and evaluated innovative interdisciplinary and capacity-building schemes. She shares insights through publications and practical workshops.

David Edwards

Dr David Edwards is an environmental social scientist with 25 years' experience in UK, Europe, Africa and South Asia. He is a member of the Senior Management Team at Forest Research, the research agency of the Forestry Commission, where he is Head of the Centre for Ecosystems, Society and Biosecurity. David manages the research programme 'Integrating research for policy and practice' which seeks to understand and enhance the impact of forest-related research upon decision-makers and land managers across the public and private sectors. He has developed and applied a range of frameworks, methods and tools to assess the cultural values associated with forests including deliberative processes with environmental artists, forest managers and local communities to create new public discourses around the cultural meanings and values associated with woodlands.

Posted In: Essential 'how-to' guides | Impact | REF2021

2 Comments

This work by LSE Impact of Social Sciences blog is licensed under a Creative Commons Attribution 3.0 Unported.