Research impact is often talked about, but how clear is it what this term really means? Kristel Alla, Wayne Hall, Harvey Whiteford, Brian Head and Carla Meurk find that academic literature discusses research impact but often without properly defining it, with academic discourses mostly drawing on bureaucratic definitions originating from the UK. The authors highlight four core elements that comprise most research impact definitions and propose a new conceptualisation of research impact relevant to health policy.
The concept of research impact pervades contemporary academic discourse. Its prominence can be viewed as part of an increasing demand for academic institutions and individual researchers to demonstrate the practice and policy implications of their work. Yet, despite its widespread usage, research impact remains an unclear and contested concept.
Search for a conceptual definition of research impact
As Australia-based researchers with an interest in translating mental health research into policy, we reviewed literature to find a conceptual definition that would guide academic inquiry into how mental health research impacts mental health policy. We chose the systematic review method because this is generally considered the “gold standard” of academic inquiry in health and medical sciences, and involves the methodical collection and critical analysis of literature in a way that facilitates quantification of findings. Thus, using a systematic review methodology enabled us to identify and quantify the prominence of different definitions and the presence and absence of research impact definitions in the literature.
Often cited, rarely defined
We reviewed 886 academic and grey literature sources (websites, online reports, and conference papers). In these, we identified 108 definitions in 83 publications. We found that only 23% – or 45 out of 200 – peer-reviewed articles to discuss research impact actually defined it. Literature that discusses and defines research impact was found to be growing, with 76% of sources having a publication date of 2011 or later.
Bureaucratic foundations
Our review identified that the concept of research impact appears to be a term of research governance rather than an academic conceptualisation. We found that 76% of the definitions either cited or were created by research organisations or funding institutions. Peer-reviewed journal articles that defined research impact often sourced definitions from grey literature.
Half of research impact definitions are from the UK
Our review also identified the United Kingdom as the epicentre of the research impact concept, contributing half (51%) of the definitions. The remainder were found in Australian (22%), other European (16%), and North American literatures (12%, combining the United States of America and Canada). These findings confirm prior research on the research impact agenda that locates its origins in the political history of the UK.
Four types of definitions
We identified four types of research impact definitions:
- as “the demonstrable contribution that excellent research makes to society and the economy” (the Research Councils United Kingdom)
- as “an effect on, change or benefit to the economy, society, culture, public policy or services, health, the environment or quality of life, beyond academia” (the Higher Education Funding Council for England and the Research Excellence Framework)
- as measurable influences in the form of quantifiable data such as citation frequency (bibliometric definitions)
- as the influences of research results on the knowledge and actions of researchers and policymakers (use-based definitions).
Four core elements of definitions
We identified four domains that underpinned the research impact concepts reviewed (Figure 1):
- contribution (the areas of research influence; e.g. economy, policies)
- avenues of impact (processes by which research could have impact; e.g. effects on knowledge, attitudes)
- change (synonyms used to describe “effects” or “benefits”)
- levels of impact (e.g. national, international).
Figure 1: Concepts that underpin research impact definitions.
Policy was expressly noted in approximately half (52%) of definitions. These definitions mostly described research impact on policy in terms of the impacts of “good-quality” academic activities and in relation to the impacts of research in changing the awareness, behaviour, or attitudes of policymakers. The definitions we reviewed tended not to describe impacts on specific policy decisions.
Critique of bureaucratic definitions
Investigating the complexity of the concept of “research impact” helps highlight underlying tensions between its uses in research governance and in dedicated academic research on knowledge translation and implementation. This tension may mirror the different practices and discourses that exist between research and policy realms, that others have noted.
One key tension is the bias, from a research perspective, of utilising research governance definitions for academic inquiry that relates to the ways these definitions unproblematically equate “impact” with “benefit”. Adopting a positive definition means that negative impacts of research would be ignored, and thus not contribute to collective understanding of the processes of knowledge translation and implementation.
A proposed academic conceptualisation of research impact
On the basis of our review, and given our specific interest in understanding the impact of mental health research on mental health policy, we propose the following definition of research impact as:
“A direct or indirect contribution of research processes or outputs that have informed (or resulted in) the development of new (mental) health policy/practices, or revisions of existing (mental) health policy/practices, at various levels of governance (international, national, state, local, organisational, health unit).”
Whilst our focus is mental health, the definition is readily adaptable to other areas of health and policy. This definition aims to capture the concepts, avenues, and levels of research impact and be inclusive of the hard-to-measure, and often non-linear, ways in which research can inform policies. It entails an examination and attention to both policy content and policy processes.
Towards conceptual clarity and academic understanding of an academic zeitgeist
Research impact is an increasingly used, but often undefined term. There are some important limitations in predominant definitions of research impact that we need to be aware of that may lead to bias if we are to advance academic understandings of how research informs policies and practice. We have proposed a conceptual definition of research impact on policy that we think is suited to various health disciplines and contexts to facilitate a better understanding of what research impact means. The next step is to apply, evaluate, and improve this definition.
This blog post is based on the authors’ article, “How do we define the policy impact of public health research? A systematic review”, published in Health Research Policy and Systems (DOI: 10.1186/s12961-017-0247-z).
Featured image credit: What? by Véronique Debord-Lazaro (licensed under a CC BY-SA 2.0 license).
Note: This article gives the views of the authors, and not the position of the LSE Impact Blog, nor of the London School of Economics. Please review our comments policy if you have any concerns on posting a comment below.
About the authors
Kristel Alla is an Affiliate Researcher with the Queensland Centre for Mental Health Research. Kristel has policy drafting and evaluation experience from working for a primary health organisation. She is completing her PhD studies at the University of Queensland. Her thesis focuses on measuring the uptake of research evidence into mental health policies.
Wayne D. Hall is a Professor at the Centre for Youth Substance Abuse Research at the University of Queensland and a Professor at the National Addiction Centre, Kings College London. He also has Visiting Professorial appointments at the London School of Hygiene and Tropical Medicine and the National Drug and Alcohol Research Centre, UNSW. He has advised the World Health Organization on: the health effects of cannabis use; the effectiveness of drug substitution treatment; the contribution of illicit drug use to the global burden of disease; and the ethical implications of genetic and neuroscience research on addiction.
Harvey A. Whiteford is a Professor of Population Mental Health at the University of Queensland, Australia and Adjunct Professor of Global Health at the Institute of Health Metrics and Evaluation, University of Washington, Seattle. He has held senior clinical and administrative positions including those of Director of Mental Health in the Queensland and Federal governments in Australia and at the World Bank in Washington DC. His expertise is in psychiatric epidemiology, mental health policy analysis and service development.
Brian W. Head is a Professor of Public Policy at the University of Queensland. He has wide experience in government and academia. His research projects are concerned with the interface between good science and good policymaking, across a range of social, economic and environmental policy areas, and issues requiring collaboration to resolve complex problems.
Carla S. Meurk is a project manager within the Queensland Forensic Mental Health Service, Queensland Health, honorary senior fellow at The University of Queensland, and Principal Researcher at Queensland Centre for Mental Health Research. Carla’s research interests are in mixed methods approaches to enhancing the translational validity of research and in improving the use of evidence in mental health and addiction services and policy.
seems too narrow – what about the wider influence on norms, attitudes, behaviour etc that lie outside policy/practice?
Thanks for writing this! I enjoyed this very much, and it got me thinking. My own approach to this is slightly simpler (I’d interested if you think this is simplistic though), and you prompted me to put pen to paper to put my own definition out there. Here’s the result – would love to your what you think:
http://www.fasttrackimpact.com/what-is-impact
Ignores the role of the Australian Research Quality Framework (RQF) which was due to be implemented in 2008 but was replaced by the ERA with the change of Aust Commonwealth Govt in Dec 2017. The RQF was very much focused on research impact and it was this policy device which in turn strongly influenced the UK government to change how it measured research performance/quality, as per the post 2008 move from the RAE to the RAF.
Like Duncan I find your proposed definition rather narrow. Longer term impacts may be linked to multiple studies and research activities that contributed to new understandings, partnerships and ways of working. These conceptual and capacity building impacts are indeed very hard to measure but still vital. If we reduce impact to direct and indirect contributions to policy and practice change we are playing into the hands of those who seek to overly instrumentalise the production and use of evidence.
I agree James and have tried to broaden it in my article here (partly in response to this blog) – do you think this works? http://www.fasttrackimpact.com/what-is-impact
I enjoyed the post and the accompanying comments. I would agree that the proposed definition of impact is too narrow in that it does not consider the impact on behaviors and practices that lie outside the remit of policy.
Mark, I think that one must be cautious when defining research impact solely in terms of ‘benefits’, as the impact produced by a particular piece of research may also have negative impacts e.g. recent work by Frey and Osborne (2017) has found that increased automation – associated with product and process innovations – may result in a reduction in as much as 47% of future employment . Furthermore, negative impacts related to increasing pollution and environmental effects, adverse health impacts or potential legal issues are often overlooked when formulating definitions of impact.