The concept of research impact represents, to a degree, a formal way of understanding the productive relationships forged between academic research and the wider world. Unsurprisingly, these relationships took on entirely new dimensions as the COVID-19 pandemic changed the world. In this post, Gemma Derrick and Julie Bayley consider how COVID-19 is potentially influencing the ongoing 2021 Research Excellence Framework cycle and its analysis of UK research impact. Pointing to the ways in which the pandemic may have differentially influenced research and impact activities, they suggest this cycle could also hold lessons for the future in relation to longstanding inequalities within the REF.

As the world continues to adapt to COVID and find ways to return to some degree of pre-pandemic normality, the UK research sector is quietly being scrutinised. The Research Excellence Framework (REF) assessment process is well underway, with panels of experts reviewing volumes of outputs, environment statements and impact case studies, with results determining the funding allocation and ‘rank’ of our institutions for the next few years.

What is perhaps overlooked, is that each submission to this periodic exercise was subject to an extended deadline because of the first wave of COVID in 2020, and then finalised and submitted during the height of the second COVID wave in early 2021.  It may be convenient to assume that the evaluation part of this process will be “business as usual”, but accommodating how the, very legitimate interruptions, caused by the COVID-19 crisis, influenced Unit of Assessment (UoA) submissions is one of the major challenges to the value of the UK’s 2021 Research Excellence Framework.

Whereas this damage is likely to be less pronounced for the assessment of the outputs, the material consequences for the university environment and sectoral impact present a risk for up to 40% of the assessment for each UoA submission (25% for Impact, and 15% for Environment). A major challenge therefore facing REF2021 panels, and research assessment processes more generally, is how to fairly and justly accommodate for variably interrupted accounts whilst maintaining commitment to creating a more equitable and diverse research culture. Furthermore, with the legacy of COVID-19 on research culture and production unlikely to be short-lived, lessons learned for this cycle provide important groundwork for accommodating for individual and organisational (including, but not restricted to gender, and disciplinary differences) disparities in future REF exercises as they continue to manifest over the next 5, or even 10 years.

COVID-19 and the REF2021 Impact criterion

The 2021 REF cycle is the second round of national research assessment with a core impact component. The 2014 impact case study database has arguably become one of the most visible expressions of research-led change, forming the basis for many research projects, commentary pieces and decision making about institutional strategies to ‘have the biggest impact’. Impact is, undoubtedly, big business, and the need to optimise submissions has generated an industry of labour, which peaked at the point of submission (March 2021).  Indeed, as expert panels review case studies, institutions are already strategically regrouping to gain ground on a future assessment for which the rules are as yet unknown.

COVID has affected every aspect of life. Within academia, it impelled a shift to fully virtual environments, with vastly increased workloads to convert in-person learning to accessible online formats and reconfigure research plans to avoid in-person contact. For impact more specifically, COVID altered the paths of impact-in-progress, with impacts suddenly halted (eg. businesses needing to stop operating), blocked (eg. the health system needing to divert all resources to the pandemic), or even lost (eg. where plans had to be shelved).  Arguably the effect of COVID was particularly potent for the Impact criterion, as the only aspect of REF dependent on the world outside the university, and a world in the grips of an unprecedented global crisis.

Are the provisions made to the REF2021 because of COVID-19 adequate?

Provision was made by REF to accommodate the challenges of the pandemic in two primary ways; (1) extensions to the submission deadline; and (2) the opportunity to provide a statement to explain how cases were affected by COVID, where such explanations were necessary for panels to make a fair judgement. Both provisions are extremely sensible, and address key practicalities of finalising a case study in this environment, but the COVID-19 crisis did not simply add a time lag to cases, nor can the full extent of mitigating circumstances be fairly communicated in a short accompanying statement. Although extensions helped the university sector prepare submissions, they did not acknowledge how COVID-19 may have fundamentally changed the narrative (and associated evidence) within the case study. We conceptualised three ‘Types’ of case study arising from these new circumstances, and have illustrated their impact trajectory in Figure 1:

A. Those unaffected by COVID-19, having been completed before or unaltered by the pandemic and needing no evaluation adjustment;

B. Those continuing through the pandemic, which required authors to react to changing circumstances, and bringing to bear evaluative challenges in compensating for evidence no longer available (corroboration uncertainty) and comparisons to the counterfactual (‘what would have been’) against the claims within the ICS.

C. Those arising because of COVID-19, such as research being used in support of public health, requiring evaluative regulation on biasing topically compelling cases. 

This newly injected variability means that just judging what is on the page without considering how cases need to be differentially mitigated will introduce inequalities into the assessment process.

In addition, the challenges associated with fairly accommodating the effect of COVID-19 on ICS is made more difficult by the practical changes to accommodate social distancing and other practical restrictions. A shift to virtual assessments – however appropriate – risks undermining the sufficiency of panel deliberations, with peer interactions necessarily altering from more naturally emerging in-person interactions. This combination of the need to evaluate more varied types of ICS, accommodate new dimensions of topicality and ‘course changed’ cases, and do so in a virtual environment, is further compounded by the potential for evaluators to introduce their own bias on ‘what counts’ based on their individual experiences of the pandemic.

We do not suggest that the REF assessment process is ignorant or dismissive of these aspects, simply that it is essential for these issues to be foregrounded in evaluation mechanisms as the assessment process is underway. Whereas the effect that these additional challenges placed on the REF2021 Impact assessment will not be known with any certainty until after the evaluation takes place (results are expected May 2022), it is important that these potential risks to fair review and evaluation legitimacy are highlighted now, not when it is too late.  Only by highlighting these risks now, can we empower current and future evaluators/evaluation panels to take reasonable action to mitigate any unintended negative effects that a COVID-blind ‘business as usual’ approach to evaluation may have on evaluation outcomes and the allocation of funding to UK HEIs for the next 5-8 years.

This post draws on the authors’ paper, The Corona-Eye: Exploring the risks of COVID-19 on fair assessments of impact for REF2021, published in Research Evaluation.

Note: This article gives the views of the authors, and not the position of the LSE Impact Blog, nor of the London School of Economics. Please review our comments policy if you have any concerns on posting a comment below.

Image Credit: Adapted from Markus Spiske via Unsplash.