Engagement activities are often noted to create clear, descriptive pathways to impact. However, the lack of standardised metrics and measures means the monitoring and evaluation of engagement activities remains difficult. Sarah Evans and Clare Deane encourage researchers to think creatively about how to measure engagement; by considering outcomes as well as outputs, ensuring indicators are truly relevant to the audience, and understanding how to properly demonstrate attribution.
Engagement is an essential part of generating impacts from research. It is also an increasingly important part of research assessment exercises and funding distribution, with HEFCE’s initial guidance on REF2021 suggesting a greater emphasis on engagement in the next round of impact case studies. At Cardiff University, we analysed our REF2014 impact case study submissions and noted how the ability to engage created a clear, descriptive pathway to impact.
A key challenge when delivering engagement is the lack of standardised metrics and measures for monitoring and evaluation. This is relevant to evaluating both engagement (the pathway, so typically a formative evaluation that can enhance practice) and impact (often more summative in nature). Researchers often want metrics that can enable them to evaluate engagement and impact quickly and easily for REF2021.
And that’s quite an unremarkable request. Successful REF impact case studies evidenced the benefits of their engagement fully. They used robust measures which told their story or theory of change. Yet monitoring and evaluation is not an easy task and the creative elements of engagement mean that using standardised metrics to measure impact is not always possible. This is because the REF is broadly defined and quantitative data and indicators are highly specific to the type of impact concerned. The route of engagement to impactful research also undoubtedly varies according to academic discipline.
It might therefore be time to think creatively about how you are going to measure your engagement, considering some of the following questions to help tailor your means of measurement appropriately:
Why are you engaging and what impact do you want to achieve?
Having a clear purpose, knowing why you’re engaging and what you want to achieve will help you measure what impact you’ve had. You need to have a direct understanding of the aim of your engagement in order to collect relevant data that will contribute to an understanding of changes resulting; i.e. what has changed, why it has changed, how changes relate to engagement, and how these changes fit into the aim of your engagement and what you want to achieve.
What outputs and outcomes would you like to see?
Although it would be easier to just measure outputs (direct, measurable results) of activities undertaken, there is a need to understand the outcomes (the changes resulting) of such activities. For example, if a learning event such as a lecture is organised it is easier to measure attendance numbers (the output), rather that the learning that occurred as a result of the event (the outcome). However, it would be more useful to know what impact the event has had (i.e. if the audience learnt anything) and to understand the quality of this not only in a binary way (e.g. an increase/decrease in learning) but from people’s experience. Think about how you can measure what is important, rather than making things that are easy to measure sound important. Articulating and capturing the outcomes and impacts of projects will provide a richer understanding of the achievements of engagement.
What measures or indicators will you use?
You need to make your measures, indicators, and evaluation activities relevant to your stakeholders/audience, and specific to what you would like to achieve. For example, if you’re engaging with policymakers, how you measure the benefits of your engagement will be completely different to when you’re engaging with school children. Speak to people, understand what measures are best for those you are working with, and work in collaboration to come up with creative indicators that will measure your impact.
How can you prove attribution?
How will you know in what ways you influenced change, and how much of it was attributable to you/your work? How do you apportion credit? By building a narrative of how change occurred, REF impact case studies allow you to evidence a causal connection which supports your contribution to outcomes. One way to help you talk about this could be through the “negative” scenario; if you/your work hadn’t been involved or had an influence, what would the outcome have looked like?
Note: This article gives the views of the authors, and not the position of the LSE Impact Blog, nor of the London School of Economics. Please review our comments policy if you have any concerns on posting a comment below.
About the authors
Sarah Evans is Engagement Officer at Cardiff University. She delivers a training programme for academics and professional services staff which aims to increase skills and confidence to deliver effective, evidenced, and high-quality engagement.
Clare Deane is Research Development Officer for the Social Sciences at Cardiff University. She currently works to support researchers to identify funding opportunities and develop and submit high-quality research proposals, and has had previous roles working to develop capacity for impact and engagement in researchers across a variety of disciplines.