Governments, funding agencies, and research organisations all over the world are now committed to measuring the impact of research beyond academic publications. Accordingly, a multidisciplinary practice called research impact assessment is rapidly developing. However, this practice remains in its formative stages and so no systematised recommendations or accepted standards to guide researchers and practitioners are currently available. Pavel Ovseiko, Paula Adam, Kathryn Graham, and Jonathan Grant introduce initial international guidelines for an effective process of research impact assessment and invite Impact Blog readers to put them into practice and share their experience, evidence, and cultural competence with the global community.
Soon after articulating the idea of scientific method in his Novum Organum Scientiarum in 1620, Francis Bacon emphasised the importance of research being “of use and practice for man’s life”. However, it was not until much more recently that governments, funding agencies, and research organisations all over the world realised the necessity of measuring research impact beyond academic publications, understanding how science works, and optimising its societal and economic impact. In response, a multidisciplinary practice of research impact assessment (RIA) has been rapidly developing and the global community of practice growing.
To help researchers and practitioners improve the process of RIA, initial guidelines for an effective process of research impact assessment have been proposed. The prima facie effectiveness of these guidelines is based on expert knowledge and practitioner experience from the International School on Research Impact Assessment (ISRIA). In a recently published paper, insights from over 450 experts and practitioners from 34 countries, who participated in the school over the last five years (2013-2017), are systematised into ten-point guidelines (Figure 1).
Figure 1: Ten-point guidelines for an effective process of research impact assessment. This figure is taken from the authors’ article “ISRIA statement: ten-point guidelines for an effective process of research impact assessment” published in Health Research Policy and Systems under a CC BY 4.0 license.
1. Analyse your context
Research and its impact are shaped by the context within which research takes place and is being assessed. That is why it is important to understand both the internal (e.g. an institution or research group) and external research environments (e.g. a country or research field).
2. Reflect continuously on your purposes
The assessment questions and methodology depend on the major purposes of RIA. Researchers and practitioners need to continuously reflect on the purposes of RIA and their relationship to the research being assessed, not least, because these can evolve over time.
3. Identify stakeholders and their needs
Research funders, research participants, researchers, research users, and research beneficiaries tend to have different expectations and intended uses of RIA. It is therefore imperative to identify and analyse stakeholders and their needs, prioritise their interests, and develop appropriate engagement strategies.
4. Engage with key stakeholders early on
Stakeholder engagement is key to value co-creation in RIA. It increases the social robustness of RIA by making it more transparent and participatory. Stakeholder engagement also makes the translation of RIA results into practice more effective and efficient by reducing the need for a separate dissemination phase.
5. Choose conceptual frameworks critically
While providing methodological guidance and analytical clarity, conceptual frameworks inevitably reduce the richness and complexity of the research being assessed. Therefore, conceptual frameworks need to be chosen critically, paying attention to the context and purpose of a given RIA exercise and to the frameworks’ limitations.
6. Use mixed methods and multi-data sources
Assessing the impact of research from the perspective of different stakeholders enhances the robustness and trustworthiness of RIA. This is best achieved by developing rich accounts of research impact using a combination of methods and a variety of data sources.
7. Select indicators and metrics responsibly
Quantitative indicators and metrics are often misused, leading to biased results and perverse incentives. To avoid such unintended consequences, quantitative indicators and metrics need to be used in a balanced way relative to the context of RIA and in support of other types of evidence.
8. Anticipate and address ethical issues and conflicts of interest
As in the case of any research and evaluation, RIA may raise ethical issues and create conflicts of interest. To maximise the social value of RIA, researchers and practitioners need to anticipate and address such ethical issues and conflicts of interest.
9. Communicate results through multiple channels
Different stakeholders often prefer different communication channels and messages tailored according to their needs and knowledge-uptake capacities. Therefore, effective translation of RIA results into practice requires a comprehensive and diversified communication strategy, including research blogs, social networks, and web feeds, among others.
10. Share your learning with the RIA community
Given that RIA is still in its formative stage, its development is sustained by empirical knowledge and practical skills of the community of practice. It is therefore imperative for researchers and practitioners to share their experience and learning with the RIA community of practice.
These international guidelines can help practitioners and policymakers in funding agencies, government, industry, charities, and academia improve the process of RIA. However, the knowledge base underpinning the guidelines has gaps in terms of geographical and scientific areas as well as stakeholder coverage and representation. Therefore, the guidelines are not exhaustive and require evaluation and continuous improvement. We invite the readers of the LSE Impact Blog to put the guidelines into practice and share their experience, evidence, and cultural competence gained through the implementation of the guidelines in new contexts with the global RIA community of practice.
This blog post is based on the authors’ article “ISRIA statement: ten-point guidelines for an effective process of research impact assessment” published in Health Research Policy and Systems (DOI: 10.1186/s12961-018-0281-5).
Note: This article gives the views of the authors, and not the position of the LSE Impact Blog, nor of the London School of Economics. Please review our comments policy if you have any concerns on posting a comment below.
About the authors
Pavel Ovseiko is the Senior Research Fellow in Health Policy and Management at the University of Oxford. He leads a multidisciplinary programme of research and policy advocacy across medical and social sciences focussing on health policy and management, gender equity and diversity, and research impact assessment. He is an alumnus of the first edition of ISRIA when it was launched in Barcelona in 2013 and subsequently a faculty member of ISRIA when it was held in Doha, Qatar in 2015 and in Favrholm, Denmark in 2017. More information about Pavel’s work can be found on his web page.
Paula Adam is the Head of Research at the Agency for Health Quality and Assessment of Catalonia (AQuAS). She leads the ISOR Group which carries out studies on the impact of health research in Spain. Paula also coordinates a commission on the accreditation and assessment of the Spanish health research institutes. She is a co-founder of ISRIA was the Director of the School when it was launched in Barcelona, Spain in 2013. More information about Paula’s work can be found on her web page.
Kathryn Graham is the Executive Director of Performance Management and Evaluation at Alberta Innovates, which is a Canada-based, publicly funded provincial research and innovation organisation. She has over 20 years of strategic evaluation experience in health care and health research and innovation. She is a co-founder of ISRIA and was the Director of the School when it was hosted in Banff, Canada in 2014. More information about Kathryn’s work can be found on her web page.
Jonathan Grant is the Vice-President/Vice-Principal (Service) at King’s College London. Service has been identified as one of five strategic priorities in King’s Strategic Vision 2029, alongside Education, Research, London, and International. Service encapsulates King’s commitment to society beyond education and research and will contribute to the realisation of its vision to make the world a better place. In addition, Jonathan is Professor of Public Policy in the Policy Institute at King’s. Jonathan was Director of the Policy Institute between February 2014 and 2017. He is a co-founder of ISRIA. More information about Jonathan’s work can be found on his web page.