In the second of our free Advice Note series on how to write Impact Case Studies for the REF, Patrick Dunleavy explains how to sift out achievements that are ‘possible’ Cases and to begin developing them successfully. The key thing here is to stay auditable and to ‘process trace’ in as much detail as you can how your research achieved external impacts.
Once you get past the over-claiming language used by the funding council (Hefce) to describe impacts, discussed in our first Advice Note, the key way to begin developing a case from your department’s achievements is to take the official demands for auditability seriously. Don’t try to blur the assessment Panel’s vision with lots of rhetorical praise for your department’s achievements. Instead play back to Hefce assessors at full throttle its official demands for chapter and verse about impacts.
The most general rule of all ‘performance audit’ in government is that at the start the auditors always ask impossible questions about assessing outputs and end-impacts. As they dig deeper they normally learn that they cannot answer these questions in intellectually defensible ways, and here honest auditors almost always regress towards process-tracing. They recognize that substantive and evaluated outcomes (such as the pink and blue boxes in the Chart in Advice Note 1) cannot in fact be assessed properly. Once this step is made, it is then natural for analysts to turn their attention instead to what can be accessed – usually a cloud of contextual indicators (such as the yellow or yellow-blue boxes in the same Chart). These multiple cues can still be sifted to try to determine a plausible (if necessarily more qualitative and perhaps vaguer) view of what is going on.
Hence a useful stage in developing an Impact Case Study early on is to try and work out which of your ‘possibles’ has the potential to help the assessors on Panels make this transition – they are going to have to do it anyway, and your job is to make it easy and comfortable for them to do so. You can best do this by trying to ‘process trace’ the basis for your claim that you achieved impact, as in my chart below.
We leave aside for the moment the thorny issue of what counts as your ‘underpinning research’ – this is an issue that Jane Tinkler and I address in detail in the third blog of this series. Instead, focus more for now on the other arrows and boxes in the Chart show, covering the ‘how’ and ‘when’ processes by which influence operated.
For Cases that claim to have influenced decisions or actors in external organizations, some key questions to ask include:
- Did people pay specifically for your work or advice? If business or NGOs turn over hard cash for knowledge, that usually indicates it was worth at least that much to them to learn about the research answers involved. For government or research foundations the case might be a tad weaker, since purely academic, ‘just in case’, ‘arse-covering’ or even ‘politically convenient’ research is not unknown. But the basic logic still works – government agencies and foundations don’t pay over money for applied research (or consultancy drawing closely on such research) without some good reason. How much your department or unit got paid is a great indicator of one aspect of significance.
If the underpinning research was financed only by a research council, an academic foundation or another funder in higher education, can you still claim its applied resonance contributed in part to receiving a grant? If your underpinning work was partly applied research, or had applied implications, can you show that this external relevance counted in your being funded? Did you have letters of support from external bodies? Did civil service or practitioner assessors involved back the bid?
Can you be precise about who exactly used your work in business, government, NGOs etc? Which sections or offices were involved, and how did they use it? At what levels of decision maker in the organizations did contacts or discussions take place? Could you show instances of your work being referenced, mentioned or discussed in relevant settings? Can you find at least some ‘echoes’ of your work in the language or framing of final decisions or choices?
Could you ask decision-makers or main external actors involved to write a brief testimonial about how your work was influential? You would need to ask them to cover not just the what, where, when questions, but to also give their view of the differences made both by the research early on and in any consequent actions taken. Could such a view also discuss the benefits from the decisions influenced or actions taken? We say more about testimonials in a later Advice Note: at this stage you just need to know, is any of this likely to be feasible.
Can you briefly offer at least some good evidence of effects? Can you trace out the gross changes made, while showing critical awareness of other influences involved, in ways that enhance the realism and plausibility of your impact claim? Can you enumerate or document any benefits and costs arising precisely from decisions or actions that were influenced, or reference some other sources that do so? If this is not feasible, is there more qualitative evidence that consequential changes were positive or valuable? If such assessments are disputed, can you nonetheless give the Panel a view of the ‘ballpark’ effects potentitally involved.
Did your contacts or involvement continue over a period of time? Can you show that your impact was not a one-off, but something that lead to continuous consultation, advice-giving or involvement? Did the research lead (even indirectly) to consultancy or contracts or other applied work? Did it perhaps lead to an appointment to a government or professional advisory committee? – one where you actively used your research knowledge (and were not just sitting there for political reasons or as A.N.Other member of the ‘great and the good’). Anything showing that the claimed impact was not just a one-off or anomalous thing speaks to the likely efficacy of the main claim.
Public engagement cases
Turning to impact cases that run more through the public debate and citizen engagement route:Can you show evidence of your research being strongly dissminated? Did you blog your research and how many people read the blogs? Did you tweet it? Did you get any press or broadcast coverage? Did any trade journals or close-to-policy journals or websites cover the work?
- Did you undertake any clear outreach activities? Did you have a road-show or launch conference or seminar? How many people came, and what kinds? How many people downloaded the podcasts or subsequent blogs? (For one recent LSE Public Policy Group conference, the room only accomodated 100 people, but 16,000 people downloaded podcasts, presentations and blogs within a couple of months). Did you make special efforts to explain your work to ‘hard to reach’ or excluded social groups?
- Did you do less attributable forms of advice or briefings – eg private meetings or workshops with outside organizations? Did you submit evidence to Parliamentary committees or other public or professional inquiries?
- Can you show evidence of your research being widely received? How many people read the blogs, downloaded research papers, cited items of your positions, retweeted things? What are the general circulation levels of magazines, blogs, news media that covered the research or event?
Across all ten of these questions, the more times that you can answer Yes, the better your case study looks. Ideally you will be able to make a case that you directly affected some decisions, actions or choices in significant external organizations, and also at the same time contributed to enhancing public debates. But the two routes may also be travelled more singly – as with private involvements that influenced other organizational actors; and with Cases that solely claim a public engagement role.
The next blog in this series begins our detailed consideration of what Hefce’s Impact Case Study forms require, section by section. We start with the issue side-stepped briefly above, anmely: What exactly does Hefce mean by ‘underpinning research’? And how exactly can you tiptoe through the minefield of rules surrounding this issue so as to successfully complete section 2 of the Impact Case Study form?
Note: This article gives the views of the author(s), and not the position of the Impact of Social Sciences blog, nor of the London School of Economics.
About the author:
Patrick Dunleavy is Professor of Political Science and Public Policy at the London School of Economics and Political Science, where he has worked since 1979. He has authored and edited numerous books on political science theory, British politics and urban politics, as well as more than 50 articles in professional journals.