As academics think about impact, they can draw on some of the lessons and strategic approaches used by civil society and campaigning groups. Andy Martin, Director of strategy consulting firm Firetail, says academics should ask themselves three questions when thinking about the impact of their research to help form a broader understanding of how their work operates beyond reductive measurables.
Most charities, NGOs and other civil society organisations are set up to change something. Achieving some kind of ‘impact’ is almost always the reason for their existence. In our experience we’ve learned that it’s hard to change the world, that measuring progress is complex, and it’s even more difficult to attribute the change you see to the work that you’ve done. ‘Impact’, especially in the worlds of politics, policy, advocacy and communications can be opaque. Yet, organisations campaigning for change recognise that they need to understand their impact as best they can.
Academics are faced with similar challenges. This may be because they are accountable to their funders, or because they are keen to understand how to maximise their impact. In academia, as in civil society, it’s critical to understand that impact isn’t just something that emerges from good work, but is something to plan for. Impact is not just a function of research excellence. It requires a new set of skills, capabilities and relationships.
To understand impact, both charities and academics need a clear view of how their work will lead to the change they want to see. To start thinking about this, there are three questions that a campaigner would ask, and that an academic thinking about impact can also use:
- How does change happen?
- Where does change happen?
- What’s my role in making change happen?
These questions are part of a ‘theory of change’ approach, which many organisations use to plan their work. A good theory of change will be iterative, and revisited to reflect what works, what doesn’t work, which assumptions were inaccurate and what needs to change. This approach isn’t about turning academics into advocates; it’s about looking at how research can have an impact, and what the causal chain of impact might look like.
1. How does change happen?
If you have a desired outcome in mind, think about the potential activities that logically map to that outcome. This mapping should be based on your view of how change happens and will, at first, be based on assumptions and imperfect information.
For example, if a charity’s ultimate goal is to engage more children in sports as a means to improve academic performance, there are a number of possible actions that it might take. One approach may be to engage in policy advocacy on the topic, another might be delivering a sports programme, a third option is to educate parents on the benefits of sports on academic performance. Each of these options has different actions associated with it, and each would have a distinct set of causal links that would likely make some progress towards the ultimate goal.
2. Where does change happen?
Looking at the landscape is a key part of any theory of change, and it can help contextualise why certain activities are successful and why others are ignored. Any landscape in which you might seek to have impact will be competitive. Those with the best solution, or the right answer, can lose out to those who are better at talking to their audience. Good civil society organisations prioritise the recipients of their output. They think about who (for want of a better word), the ‘customers’ of their work are. This might be the people reading policy research, the audience of their advocacy work, or the recipients of programmes, or the general public.
We spend a lot of time helping our clients to think about their audiences and their environment. What does the political and environmental landscape look like? Who are your ‘customers’ already looking to for guidance? What are the messages you are competing with? How do these groups behave, and what will compel a change in behaviour? What are the constraints facing your ‘customers’? Who has the power to implement change? These questions are relevant to anyone looking to have an impact with policy and research questions.
Going back to our example of engaging more children in sports, let’s consider the context. There may be a scenario in which the political appetite for mandating more sports in schools is strong, it’s an issue that has popular appeal, and the economic conditions make it possible to implement. In this context, policymakers might be receptive, policy research work may be wildly successful, and an organisation might attribute their success to a job well done. The organisation might repeat their efforts for a different project under different circumstances. In a scenario where the conditions are unfavourable, the same quality inputs could have minimal results.
Haphazard work can have an impact in favourable conditions and impeccable work can fail due to tough circumstances. Separating how much of your impact is environmental is highly subjective, but essential to learning.
3. What is my role in making change happen?
Once they have a view on how change happens and the landscape for that change, strategic charities and NGOs plan their interventions to have maximum impact. Based on what they know, they ask themselves “What can we do to cause the change we want to see in the world?”
In our work with academics, we have seen simple changes in approach lead to greatly increased impact. It can be as simple as making sure the right officials see your work, or having a team deciding to focus on EU-level rather than national-level stakeholders. It might mean timing publications to coincide with major events, like the G8 or the reading of a bill in Parliament. A view of how audiences engage with research will lead some to publish briefings for politicians that are short and to the point, whilst research for officials is fully referenced and cited. Work for local decision-makers can be customised with local data. It often means developing a clear understanding of how the media landscape works, what it takes to be credible in that world, and how an academic’s work in the media connects to political change.
But without knowing how change happens, what the landscape and the audience looks like, and what your role could be, it’s difficult to be effective. Much of the current debate about ‘impact’ appears to focus on the identifiable, measurable and verifiable. This feels reductive, especially when the bigger picture often tells a more important story.
A ‘theory of change’ is a way of describing this bigger picture.
Answering these three questions is a starting point for thinking about impact in a more strategic way. It’s an approach that many civil society organisations use and find useful, and something academics are beginning to use to provide context and focus to their work.
Note: This article gives the views of the author, and not the position of the Impact of Social Science blog, nor of the London School of Economics. Please review our Comments Policy if you have any concerns on posting a comment below.
Andy Martin is the Director of Firetail, a strategy consulting firm that with civil society organisations. Since founding Firetail he has worked as a senior advisor to leaders of charities, NGOs and social enterprises including Comic Relief, Oxfam, Anthony Nolan, World Vision, the Open Data Institute and the Bill and Melinda Gates Foundation. He was previously Head of Strategy at Cancer Research UK, the UK’s largest charity and spent the early part of his career in commercial strategy consulting.
Two complications for the worried academic in this context:
(1) Academic theories of policy change tend to point to long periods before anything major happens – often describing change over decades, or at least longer than impact case studies are designed for
(2) They also tend to portray policy change as multi-causal, making it difficult (if not impossible) to make any great claims about one’s part in that change.
In other words, one consequence of seeing the big picture is that our part in it seems tiny.
Thanks for this insightful post!
@Paul Cairney: I agree entirely,and I’d also probably suggest it’s far, far worse than that!
As you write, the attribution of policy impact and political change is hugely complex, and not something that can be reduced to a formula. (And nor, if you think about it, would you want it to be). Change is slow, non-linear, multi-causal and hugely subjective.
Because this is all so opaque and non-verifiable, in our experience, understanding the context really matters, both in planning and in evaluation.
From an evaluation perspective, I’ve found that individual instances of quantifiable, verifiable impact are almost always less useful and insightful than holistic, qualitative, contextualised evaluations of an organisation’s overall place in a policy debate.
The article “The elusive craft of evaluating advocacy” by the Stanford Social Innovation Review is interesting on this topic.
Yes, so the issue now is how the REF panels will deal with this insight. Many people on panels have made the right noises about seeing academic impact in the round but, then again, many have said that they are going to read every academic submission too.
Great article, and theories of change reflected in the application process of funders, sometimes prevents innovation seeing the light of day. Even subtle changes in approach by applicants can be dismissed for the benefit of structure and process not impact.
Andy, is this ‘theory of change’ approach something your consultancy has developed on its own, or are there key texts or thinkers curious readers could track down if they were interested?
While I agree that pragmatic theorising can be helpful (I agree with Kurt Lewin’s claim that ‘…there’s nothing so practical as a good theory’ – key is what makes it good?) there are influential ways of working adopted by Lewin that he called ‘action research’. In many respects this aligns with current ideas about co-production – doing this means you rely less on just theory (ie how you *think* the world works) and continually test it with those whose job it is the actually try and change it – ie the ‘policymaker’. Generating a theory of change is all well and good but you need practices of co-production & discovery about the ‘problem’ and ‘solutions’ to have a better chance of informing policy effdctively. At least this is the approach we are taking in our new professional Masters programmes for mobilising science or engineering for public policy at UCL STEaPP 😉