The impact of academic research, particularly on policy and the private sector, is an increasingly important component of research assessment exercises and funding distribution. However, Duncan Green argues that the way many researchers think about their impact continues to be pretty rudimentary. A lack of understanding of who key decision-makers are, a less-than-agile response to real-world events, and difficulties in attributing credit are all hampering progress in this area. Looking at how impact is measured by aid agencies, there is much academics could learn from their monitoring, evaluation and learning teams.
As someone who works for both Oxfam and the LSE, I often get roped in to discuss how research can have more impact on “practitioners” and policy. This is a big deal in academia; the UK government runs a periodic Research Excellence Framework (REF) exercise, which allocates funds for university research on the basis both of their academic quality and their impact. Impact accounted for 20% last time around, in 2014, and may have an even greater weighting in the next exercise, in 2021. That means hundreds of millions of quid are at stake, so universities are trying to get better at achieving and measuring impact (or at least looking like they are!) as they start to prepare for the next round of submissions.
And they have a way to go, based on my recent experience of listening to pitches from a range of researchers. Some interesting patterns emerged, both in terms of weaknesses and ways to address them.
First up, many academics fall into the classic mistake of equating frenetic activity with impact. Long lists of meetings attended, papers published, speeches given, etc., do not impact make. I think they should start from a different place – how would you set about convincing an intelligent, fair-minded sceptic of the impact of your research?
That might be a challenge because the way a lot of researchers think about their impact seems pretty rudimentary. Finish the research, publish the papers, run a few seminars, produce a policy brief and (at the less stuffy end) bash out some social media, and voila! Unfortunately, “blog it and they will come” is often not a very convincing influencing strategy.
A popular alternative in REF submissions is the adviser/consultancy model – “the UN/Government asked me to present/be on a panel/draft their guidelines – now that’s impact”. A bit more convincing, but both approaches would benefit from a more explicit theory of change, running through some of the following issues:
Stakeholder and power mapping
- For the change you are advocating, who are the likely champions, drivers and, undecideds who you can seek to win over?
- Who has the power over the decisions you are trying to influence? They are probably going to be your main targets.
- Who/what in turn influences those targets? Is it evidence (if so, what kind?) or something else entirely, like the identity of the messenger?
- How do you get targets to be aware of and interested in your research in the first place (since both decision-makers and practitioners live in the land of TL;DR)? Have you tried to involve them in it, by asking them to be on a reference group, comment on drafts, or be interviewed for it, for example? Much better than just adding your paper to their reading pile.
Dynamics
Is the change you seek likely to be smooth or spiky? Some changes are largely crisis-driven, in which case your influencing strategy will have to try to get ready in advance, identify and make the most of crises or other “windows of opportunity”. As Pasteur said, “fortune favours the prepared mind”. This is antithetical to the normal rhythm of research – a steady, high-pressure grind of papers and presentations, largely oblivious to events out there in the world.
Insider vs Outsider strategy
Insiders prize being “in the room”, and see themselves as deeply engaged in policy formulation. Outsiders are more interested in constructing alliances, working with (and providing ammunition for) activists and civil society organisations, and talking to the media. But both need to think through their influencing strategies. On the insider route, are the people in this particular room really the people making the decisions? Which invitations do you say no to (if only to avoid death by consultation)? For outsiders, what are the alliances you need to make in order to reach and influence the decision-makers on this particular issue? In both cases, what kind of research (both in content and presentation) is most likely to get people’s attention?
Attribution
Both approaches struggle with attribution. According to one insider “the ministry is a black box – you’re invited to speak. They say a polite thank you. That’s it.” How do you know if you changed anything? The challenges with outsider strategies are different – in an alliance of a dozen CSOs, think tanks and universities, even if some change has been effected, how do you apportion credit?
The efforts at attribution are currently pretty crude: count the citations and if you think your research has helped someone, you beg them for an email saying just how brilliant you are and how much impact you have had. Better than nothing, but pretty dubious as a sole source of evidence (right up there with job references in terms of rigour….).
There’s a lot that academics could learn from the MEL (monitoring, evaluation and learning) teams in the aid agencies here – they have some incredibly smart people working on this. I’ve seen Oxfam’s Effective Reviews really invest in trying to measure impact by constructing a plausible control group for comparison, through propensity score matching, or comparing and ranking different causal factors through process tracing. The academics I’ve talked to seemed entirely unaware of these kinds of methods. Given the multi-million pound benefits of getting it right, there may be a case for universities to outsource (and pay for) this either to “impact partners” such as NGOs with more experience in the area, or to third-party specialist organisations that could accompany researchers in design for impact, and then measure the results more rigorously.
Then there’s communications. The default tone is neutral and cautious – goals were to “shift” this or “support” that without ever specifying why or towards what. That feels more like avoiding mistakes than communicating a message. One alternative would be to come up with a clear mission statement – “we use evidence to change public opinion so that X happens”. If that’s too close to a credibility-damaging move into overt campaigning, academics should at least be able to clearly identify their USP (unique selling point) – e.g. “I’m the only person in the UK who knows what is going on in these particular far-off places”, and clearly state what their research is about in such a way that normal people and REF panels can understand.
That sounds like an extended moan, but perhaps most striking in the conversations I’ve had is how much impact appears to be taking place anyway – I came away with the impression that a bit more analysis and explicit discussion of assumptions and influencing strategies could be an incredibly productive exercise for universities seeking to turn research into real impact.
This blog post is an edited version of that which originally appeared on the author’s From Poverty to Power blog and is reposted here with permission. A number of counterpoint comments have been posted underneath the original version, and offer an interesting alternative perspective. The exchanges that followed have been written about in Duncan’s more recent blog post.
Featured image credit: Heartland Archery | Winnipeg by Justin Ladia (licensed under a CC BY 2.0 license).
Note: This article gives the views of the author, and not the position of the LSE Impact Blog, nor of the London School of Economics. Please review our comments policy if you have any concerns on posting a comment below.
About the author
Duncan Green is Senior Strategic Adviser for Oxfam GB and Professor in Practice in the Department of International Development at LSE. He runs the From Poverty to Power blog and is author of the book How Change Happens. He can be found on twitter @fp2p.
I think this is what we have advising our researchers – impact has to be designed in at the start, and not via a reflective look back on what impact may or may not have resulted.
Certainly research has to test the parameters of the questions being asked, but at the same time there has to be a deliberate process of guiding the research towards some pre-identified notion of where impact is likely.
Duncan gives a very good analysis of the way impact measuring is done by many research institutions and provides some valuable ideas how they could improve their impact and impact measurement for policy oriented research which focuses on short term objectives and directly influencing decision makers. However, this is not the mainstream research. Many research activities are focusing on a more indirect influence of policy by changing general world views. This mainstream research is not focused on short term objectives but on long term objectives to improve our understanding of social behaviour. Much of this research is first discussed and refined with and among peers before it gets accepted and appreciated by outsiders and by decision makers that may be biased by previous (wrong) understanding. The findings of this research may be even contradictory to the results of short term objective driven research and finally have a much greater impact in influencing decisions of larger groups of decision makers. It is impossible to measure the impact of this kind of research on the short term and very hard to measure the impact of this kind of research on the long term in another way than a kind of anecdotic impressionistic way.
It is a very nice post I was interested to know about researchers’ point of view on research impact at any stage of doing research or after say 10 years.
I really would like to read a post about how to present qualitative data using software (programs) and how that is effective in representing the data in the thesis.
Thank you,
Wafaa
There are too much political closed mindedness and too little willingness for examining the new discoveries in a sincere way. Nobody REALLY want to know how to do it better because those already doing it would either loose their jobs or have to begin again to understand how it works. No amount of computing can change that until we are organized by people or robots who have unbiased minds.
Duncan, that’s a useful and insightful post. Do you have thoughts on how universities should fund MEL?
There’s no extra funding in the system that I can identify. Monitoring needs to be done either long after the research (grant) ends or at a low level for many years. UK funders’ grants for generating impact are precisely that and don’t leave much space for monitoring.
Even with funding, it’s debatable whether an evaluation person or the researcher is best place to ask the right questions of various stakeholders.
Andrew
Thank you very much for this very interesting article and in particular to the list of questions referring to stakeholders and power.
The author , in the conclusions recommends: “bit more analysis and explicit discussion of assumptions”
.I agree, but I wonder why an academic working for Oxfam ( a champion in the analysis and struggle against inequality) is not referring explicitely to asymmetries of power and inequality, or better to polarisation, as assumptions to be taken into consideration both .in development as in research activities
Massimo Rossi,
The author presents a very interesting list of questions related to stakehoders and power and in the conclusions recommends: “.a bit more analysis and explicit discussion of assumptions”
I wonder why, concerning the assumptions and the power, only reference is made to decsion makers and drivers and no reference is made to asymmetries of power and to polarisation, as assummtions to be taken into consideration both by researchers and by field workers.
We are seriously not serious about making the sort of impact you are talking about. We research, we teach and we influence. Why should researchers trying to prove to the world that they are making inpact? if they have no intention to be held accountable for the negative impacts their research has generated, why should they be so keen to claim the good impact after all.
There is a reason that there are labour division between think tanks, academics and practitioners. If robots are taking up the jobs of others are considered disruptive, academic robots doing other people’s work is also disruptive.