We are in danger of suffocation by the ‘impact agenda’ and risk losing sight of the original value of research, writes Liz Allen, who argues that negative findings and research failures are essential parts in the journey to increasing academic knowledge.
Funders have a long history of monitoring and evaluating the schemes and programmes that they run. This involves trying to elucidate the key achievements of those who have been funded in addition to investigating what has and hasn’t worked.
Every funder has goals that guide the choices it makes. When we a make a choice about what research to fund we take a risk – we weigh up information about the applicant, their idea, and the environment in which they intend to carry out their research. Funders are keen to understand how they can be more effective, so understanding the impact of its choices is vital. Over the last decade, the word impact has been somewhat hijacked with the emergence of the ‘impact agenda’ within the research funding world. This has led to all kinds of curious takes on how the outputs and outcomes of research should be measured, illustrated or explained.
Research moves, for the most part, in incremental steps, with each new discovery laying the foundations for future work. When you map out the route of a particular strand of research, you rarely (if ever) find a linear path, with a single funder or researcher. Some actors may have been more instrumental than others, but it’s not easy to tease out the precise contribution of each. Indeed, the true value of the different contributions may change as results become clear.
In the biosciences, the time elapsing between initial research and its tangible impact on health can be great. Recent analysis has estimated that, for a selection of cardiovascular-related medical interventions, the time lag between key research and impact on health was around 17 years. It took the reproductive biologist Robert Edwards nearly two decades from being able to mature human eggs in a laboratory to the delivery of the first in-vitro fertilisation (IVF) baby in 1978; it took nearly three decades from the discovery of monoclonal antibodies to their widespread adoption in medicine2. An essential part of evaluation is learning how and when research leads to impact and how we might make the process of doing research more efficient.
The impact of putting our best foot forward
We have grown accustomed to using indicators such as bibliometrics to tell us if research is moving in the right direction. We know that knowledge production is important in its own right and its re-use in future research and in policy and practice is an indication as to its value. We want to know about career progression. We want to know about new research and the leverage of funding and a host of different things. And different funders place their emphasis on different things – both when choosing to fund a piece of research and when considering the outputs of the research they have chosen to fund.
The case study or narrative story of a piece of research has become valuable evaluation currency in recent years – write a case study, make a case, job done. While stories are powerful and do explain the progression of a piece of research, as with metrics they need to be used appropriately; a ‘case’ is exactly that, a case and the choice of which pieces of research make it to case study status is typically highly selective. What you don’t see much of is analysis across case studies – looking for themes and common characteristics that might have led to a successful outcome, distilling what’s important and considering what works and what doesn’t.
While we are busy putting our best foot forward, we are in danger of moving the impact agenda to a place where it is hard to recognise the value of actually doing the research and negative findings or research failures, which are essential aspects of both science and funding.
Moving beyond impact
We all understand the drive to allocate resources in the best way. Difficult choices have to be made – especially in times of austerity. Policy-makers need tools to provide the evidence to assist them in their decision-making. The problem is, and this argument has been well-versed in this blog, that it is difficult and complex to design a set of tools that will deliver measures of impact that are proportionate, economical to use and capable of reflecting the realities of research within an appropriate timeframe.
But we (the whole research community) need to pragmatic and move the field of impact assessment forward. Many of the indicators used to measure research progression and impact, while fairly robust in the aggregate and for the mainstream, do have flaws. We need to remember the definition of the word indicator. We need to develop better qualitative ‘tools’ to complement the quantitative indicators of impact. And we should be using our assets to their best effect and in the research funding industry our assets are our researchers. We should encourage and reward openness and transparency and the sharing of the trials, tribulations and products of research. On the other side of the coin, we should be sharing information on what works and what doesn’t in research. Researchers and funders should never be ‘them and us’ – we are on the same team.
We need to move ourselves beyond a research impact agenda where the focus encourages us to make the case and put our best feet forward. Instead we should concentrate on being transparent about, and learning from, all aspects of the journey.