Lucie Rychetnik and Robyn Newson were part of a research group examining the ‘real-world’ impacts of health intervention research. Using an impact assessment scoring system, they found a wide range of possible impacts. They also found local contextual and organisational factors, and unpredictable windows of opportunity were as important as the skills of individual researchers and the quality of their research.
Increasingly, in both the health sector and beyond, we have expectations that those making important decisions will consider and apply research evidence to inform what they do – be that public policy, program and health service delivery, or individual practice. The underlying logic for this expectation being of course that good research generates important and valuable knowledge – and that when this knowledge is applied it leads to better and/or more efficient policy and program decisions.
Concurrently, many now have the also growing expectation that publicly funded research will offer some practical ‘real-world’ relevance and value – beyond that of merely generating interesting insights. Thus research funders, research institutions and researchers themselves increasingly expect that research findings will, ultimately at least, contribute towards social, material, health or technological improvements. Indeed much of the current discourse and analysis of what we mean by research ‘impacts’ have been reflected in the LSE Impact blog. As a result, indicators of post-research impacts have been considered and trialled as measures of the overall merits and value of funded research. At the same time, the epistemological and practical implications of this have also been challenged.
At the University of Sydney, Australia, a group of us recently completed a study that examined the ‘real-world’ policy and practice impacts of health intervention research which had been funded (project grants) between 2003-2007 by the Australian National Health and Medical Research Council (NHMRC). As part of this study we trialled a post-research ‘impact assessment’ tool, which was adapted from the literature. It incorporated impacts in the form of scholarly outputs, translational outputs, and policy and practice impacts – plus a scoring system for rating impacts along four dimensions: corroboration; attribution, reach, and importance. Further, as well as investigating ‘whether’ impacts had occurred, we also examined ‘how’ they occurred.
Image credit: Florida State Road Department Midtown Interchange (Wikimedia Public Domain)
In this study, we defined policy or practice impacts as ‘demonstrable changes, or benefits to products, processes, policies, and/or practices, that occur after a research project has concluded.’ We looked for impacts that were concrete, measurable changes in policy or practice such as a new government policy, change in organizational or clinical practice, health education campaign, or related new funding that could be attributed to the study intervention. This could also include stopping or changing existing interventions following demonstration of intervention ineffectiveness.
Our focus was on intervention research as this is more likely to have direct implications for health policy and practice. We used a broad definition of intervention research that included interventions from a wide variety of health disciplines, delivered in clinical and community settings. We excluded pharmaceutical interventions as these require authorization by Australia’s Therapeutic Goods Administration via a standardised formal process.
Of the 50 eligible project grants included in our sample, 38% had some demonstrable impacts on policy and practice by the time of our data collection in 2012/2013. A wide range of different types of impacts were recorded, and the assessed level of impact was evenly distributed across our three categories of low, medium and high.
We designed our impact assessment scoring system to take account of commonly identified challenges with respect to the attribution of impacts to particular studies, and the importance of seeking evidence to corroborate the reported impacts. Where the supporting information was limited or absent, we found a greater degree of variability in the impact assessment scores given by our expert panel.
In terms of ‘how’ impacts occurred, our findings revealed this to be a complex interplay between what researchers themselves do and a wide range of other factors that were largely outside the researchers’ control. Researchers’ perceptions about the implications (potential influence) of their findings sometimes determined whether and what steps they took to disseminate these findings. Some vigorously pursued dissemination opportunities if they felt their findings warranted such action, and researchers’ ability to recognise how their research could be useful, and then how to engage with the post-research context were important. (Figure 1) Ultimately however, each researcher only had partial influence on the policy and practice impacts of their research. As has been suggested elsewhere, research impacts were a product of the whole system and could not be produced by the researchers alone.
In Figure 2 we summarise, for the purpose of discussion and further inquiry, the potential implication of our findings for funding bodies, research institutions and researchers. Most of these implications however, have associated risks and caveats. For example, allocating funds to research that supports existing lines of enquiry can build a momentum for policy and practice impact, but funding portfolios must always include funds dedicated to innovation, where the implications for post-research impacts may not be so readily apparent. In addition, grant assessment processes need not favour only researchers with established track-records of translational activities if early career researchers can also demonstrate prior policy and practice networks (former colleagues or other collaborations) and relevant translation skills (there are a growing number of Knowledge Translation courses aimed at researchers).
Of course the bigger questions remain about the long-term value of governments and research funding agencies measuring the impacts of research. Ideally, if ‘real world’ impacts are deemed to be important, then funding agencies would want to predict these – and invest their resources accordingly. But is that even possible? We know from studies such as ours, and the wider KTE (knowledge translation and exchange) literature, the influence of research on important ‘real world’ decisions is as likely to be associated with local contextual and organisational factors, and unpredictable windows of opportunity, as it is with the skills of individual researchers and the quality of their research.
And finally, we are yet to see what, if any, will be the impact of the growing focus on post-research impacts. In their recent and very useful report ‘Mobilising Knowledge to Improve UK Health Care’, Davies and colleagues reviewed the international literature and the knowledge mobilisation activities of agencies working across health, education and social care, to draw out the lessons and implications for current action and future research. They concluded that further investigation of the merit and value of approaches for assessing research use and impact was an important area for future development.
We know that as with any system, an introduction of new measures, incentives and rewards introduces new feedback loops, and inevitably changes the way that system works. It is also likely to change the activities of those working within the system, and the products the system will generate. If we are indeed seeking to shift the priorities and focus of researchers, and thus the nature of the research being done, we should certainly monitor if this is indeed a good thing.
Note: This article gives the views of the author, and not the position of the Impact of Social Science blog, nor of the London School of Economics. Please review our Comments Policy if you have any concerns on posting a comment below.
Lucie Rychetnik is Director, Research in Knowledge Synthesis and Exchange at the Sax Institute (http://www.saxinstitute.org.au/); Associate Professor (Translation Research) at School of Medicine Sydney, University of Notre Dame; and Honorary Associate Professor at the School of Public Health at the University of Sydney. Email: email@example.com Twitter: @LucieRychetnik