LSE - Small Logo
LSE - Small Logo

Julie Bayley

Kieran Fenby-Hulse

Chris Hewson

Anne Jolly

June 14th, 2021

Industry not harvest: Principles to minimise collateral damage in impact assessment at scale

2 comments | 73 shares

Estimated reading time: 7 minutes

Julie Bayley

Kieran Fenby-Hulse

Chris Hewson

Anne Jolly

June 14th, 2021

Industry not harvest: Principles to minimise collateral damage in impact assessment at scale

2 comments | 73 shares

Estimated reading time: 7 minutes

The recent institutional submissions and conclusion of the first phase of the REF, coupled with the announcement of a wide-ranging review of research assessment in the UK, has provided space for renewed thinking on the state of research assessment. In this post, Julie Bayley, Kieran Fenby-Hulse, Chris Hewson and Anne Jolly, present reflections on the wider systemic effects of research and impact assessment within higher education institutions during the most recent round of the REF and discuss how principles derived from these observations might inform an approach to research assessment that is more inclusive, consistent and reduces unintended consequences.


As the UK closes the curtains on the Research Excellence Framework 2021 (REF2021) and embarks on another round of consultation, there is little doubt that, whatever the outcome, the expectation remains that research should be shown to be delivering impact. If anything, this expectation is only intensifying. Fuelled by the stated success of REF 2014, the appetite for impact assessment also appears – at least superficially – to be increasing internationally, albeit largely stopping short of mirroring a fully formalised REF-type model. Within this context, the UK’s Future Research Assessment Programme was recently announced, with a remit to explore revised or alternative approaches. Everything is on the table, so we are told, and the programme sensibly includes the convening of an external body of international advisors to cast their, hopefully less jaded eyes upon proceedings.

with impact-as-assessment models poised to expand globally, the experience of ‘early adopter’ countries (such as the UK) must be critically interrogated in terms of the strategic, operational and ethical challenges such frameworks usher forth

At this pivotal moment, with impact-as-assessment models poised to expand globally, the experience of ‘early adopter’ countries (such as the UK) must be critically interrogated in terms of the strategic, operational and ethical challenges such frameworks usher forth. The sector is already vocal, highlighting concerns over rule complexity, alongside a recognition of the pressures to deliver impact by an already burnt out and anxious workforce. The utility of the REF in demonstrating higher education’s contribution to society, and providing a defensible basis for funding allocation, must therefore be balanced with the unintended consequences for a research sector under extreme pressure. Below we reflect on three key areas of collateral damage that will be recognisable to those on the ‘factory floor’, and from these we propose a set of principles for research assessment designers to build into future governance and accountability processes.

1.     Resource diversion and cannibalisation

There are well documented costs to the REF, with the 2014 exercise, despite repeated promises to address the burden of bureaucracy, reputedly costing £250 million; many times higher than previous RAE cycles. Nevertheless, whilst critics regard REF as a ‘waste of time’ a post-2014 review – chaired by Lord Stern – found that for some, the “benefits outweigh the costs” and the sector would be “poorer without it”. These macro-level economic debates invariably obscure the implications for institutional support practices, in the context of finite institutional budgets.

The REF is a cyclical assessment, currently with a seven-yearly (or so) timestamp. As a consequence, effort intensifies at key points, creating resourcing peaks and troughs, with impact brought into sharp relief at different points of the cycle. This is compounded by the existence of cross-cutting and concurrent agendas – such as the Teaching Excellence Framework and Knowledge Exchange Framework – each bringing to bear competing requirements and further pressure on research and enterprise offices. At smaller institutions, these pressures often necessitate the diversion of resources – staff reassignments, ringfencing of tight budgets and redistributed administrative support – towards functions directly in support of REF administration.

Unsurprisingly, resource demands were particularly intense in the latter months of REF2021, with a plethora of staff conscripted to deliver the final submission. Similarly, a reserve army of short-term posts were hastily created to amass evidence and finalise case studies. A recent ARMA survey showed that a significant volume of impact-related posts (72%) were formally due to end at the completion of this cycle. Such staff turnover is inimical to the purported aims of the REF – to embed impact into research culture – and is clearly also inefficient on its own terms. In the short term, REF preparations cause research and enterprise resources to be directed away from wider research support and development activities. In the longer term, precarious contracts mean impact expertise is lost rather than developed, with local impact cultures subject to a constant state of reset.

2.     Ingrained apprehension and fearfulness

In January 2020, David Sweeney asked institutions to be bold when developing impact case studies. However, the financial and reputational penalties for ‘getting it wrong’ are high, with results (hence income and ranking) locked in until the next assessment cycle. Mechanisms presented as opportunities for institutional liberty in decision making – choice over case studies selected, in some cases ‘indicative’ rather than concrete rules, non-prescribed evidence options – instead breed anxiety and apprehension, as institutions and (often) inexperienced staff wrestle with the spectre of accidental eligibility breaches.

For example, changes between the 2014 and 2021 supposedly proffered a greater recognition of public engagement and impact on teaching, yet also introduced tensions in judgement about how much engagement and what would realistically count in teaching. In parallel, despite impact being now well recognised as an often non-linear, messy, and iterative process, assessment templates demand a sequential, dissected yet smooth narrative from research-to-impact, magnifying expectations of chronologies which don’t mirror the research process or academic careers.

At a more fundamental level, institutional concerns about accidental non-compliance or a poor-quality submission not only injects fear in the construction of case studies, but can also fuel selective and strategic decisions that ultimately stifle research regarded as unlikely to deliver a high impact yield. Ultimately the technicalities of eligibility, and the base principle that impact is ‘owned’ by institutions, makes boldness a high-risk strategy, and one which can so often leave impact stories muted.

3.     The commodification of allies

Impact is a team sport, yet REF assessments neutralise accounts of the contributions of those outside the unit of assessment (UoA) research team, as well as professional services staff and others deemed to be not research active (most ironically PhD students). Leaving disciplinary differences aside, REF impact is invariably an imperfect judgement of the overall balance of inputs and outputs. Accordingly, the effect on research culture is uneven and problematic; those named at the head of wider collaborative impact endeavours become star players, others are rendered invisible. Those with secure and long-standing positions take precedence over those with short-term, precarious contracts. This is not new, and is an extension of problematic leadership cultures (with their attendant EDI deficits) that the current Chief Executive of UKRI is seeking to address, alongside ongoing challenges for early career researchers to establish a foothold in academia.

The centring of the researcher to the exclusion of others is a structural, rather than incidental, feature of REF, wherein impacts are ‘owned’ by academia.

The centring of the researcher to the exclusion of others is a structural, rather than incidental, feature of REF, wherein impacts are ‘owned’ by academia. It also has the effect of commodifying and instrumentalising relationships, placing an unrewarded burden on those outside academia to provide evidence for case studies they ultimately do not benefit from. The gathering of evidence, sometimes portrayed as a simple request for feedback, is in practice imbued with fundamental issues around not only stakeholder effort, but sensitivities related to data protection, confidentiality and ethics, which can undermine rather than facilitate the development of long term relationships. Requiring institutions to gather their own evidence arguably says more about a wider systemic move to push bureaucracy back onto universities, than it does about optimising impact.

Healthier approaches to research assessment

Engagement and knowledge exchange may be an increasing mainstay of academia, but the requirement to gather evidence of social change which is sufficiently large and optimally curated to attract funding, is an industry of its own. The issues outlined above reflect areas of concern from the ‘factory floor’, and point to the detrimental effects on institutional health, stakeholder engagement, and wider research culture that emerge from a peculiarly uneven form of impact assessment. Unfortunately, it is immensely difficult to provide unequivocal evidence of these forms of collateral damage, not least as there is little institutional appetite to shine a light on such dark corners of practice, especially amidst post-REF flux in roles and leadership.

Ultimately, the combination of finite resources, institutional survivalism and interpretative rules creates a petri dish where damage can intentionally or unintentionally thrive. However, as we refresh our thinking, we now have the perfect opportunity to draw from good practice emerging elsewhere in the sector, such as inclusive authorship strategies (e.g. CREDIT), responsible evaluation (e.g. SCOPE) and drives to unveil broader contributions (e.g., the Hidden REF). Moreover, whilst solutions to the issues outlined are beyond the scope of this piece, and potential remedies will themselves require checking for unintentional consequences, there are some clear principles for a more equitable, effective and less depleting system:

  1. We must recognise that impact assessment is an industry, not a harvesting of naturally occurring effects, and that this has resourcing implications which are felt unevenly across the sector
  2. A clearer understanding is required of the disenfranchising effects felt by those in the broader research ecosystem, particularly individuals not portrayed as impact leaders within case study narratives
  3. The sector must develop a clearer understanding of the risks inherent in instrumentalising and commodifying stakeholder relationships, and how this sits in contradiction to other parts of the research ecosystem (e.g., KEF)

If we are able to anchor our planning within these three principles before we consider the inherent benefits and optimal processes of research assessment, we can avoid some of the damage that is currently ‘baked in’ to the system. Embedded and equitable approaches bring advantages for institutional memory and impact literacy, and underpin fairer expectations regarding the types of impact that certain types of research might generate . Moreover, in widening these discussions and not simply talking to ourselves, we can help maximise societal benefit in collaboration with, rather than at the expense of the academic community as a whole.

 


Note: This review gives the views of the authors, and not the position of the LSE Impact Blog, or of the London School of Economics.

Image Credit: Adapted from Jean Wimmerlin via Unsplash. 


Print Friendly, PDF & Email

About the author

Julie Bayley

Dr Julie Bayley is Director of Research Impact Development at the University of Lincoln (UK) and Director of the Lincoln Impact Literacy Institute (LILI). She collaborates nationally and internationally on the development of impact literacy alongside working as Emerald Publishing’s Impact Literacy Advisor, Director of Qualifications for the Association of Research Managers and Administrators (ARMA) and works with funders to strengthen impact. Julie is a Chartered Health Psychologist with a PhD in Health Psychology and Impact and a patient advocate in vascular health.

Kieran Fenby-Hulse

Dr Kieran Fenby-Hulse is Research Fellow (Research Policy and Impact) at Teesside University. Kieran specialises in research communication, engagement and impact and undertakes research on the intersection between cultural policy, entertainment, and costal regeneration and seaside towns. His recent publications include a study of the experiences of LGBTQ doctoral researchers and an edited collection on Research Impact and the Early Career Researcher. Kieran is also the managing editor for the Journal of Research Management and Administration.

Chris Hewson

Dr Chris Hewson is Faculty Research Impact Manager (Social Sciences) at the University of York, having previously worked in impact support roles at the Universities of Manchester, Salford and Sheffield. Chris provides expert assistance to York’s social sciences departments regarding all aspects of research impact, knowledge exchange and business engagement, including but not limited to funding and the REF. He also manages York's Economic and Social Research Council Impact Accelerator Account (ESRC IAA), and the York Policy Engine. Chris’s PhD examined the development of community radio and television services in the face of social and regulatory change.

Anne Jolly

Dr Anne Jolly is Impact Development Officer at the University of Lincoln, having previously held scientific roles with NERC, Infoterra and the Met Office, and in Data Protection at Lincolnshire Police and the University. Anne is based in the central Research and Enterprise office, and works institution-wide to support impact, most recently in support of REF.

Posted In: REF2021

2 Comments