Following submissions to the REF in March, Pauline Addis and Anthony Atkin reflect on the experience of the professional service staff in producing impact case studies. Highlighting two key challenges faced across the sector, they suggest future assessments would benefit from more stable guidelines.
At the end of March 2021, the sigh of relief from across the UK HE sector was almost audible. After years of work, months of polishing and weeks of stress, REF2021 was over and those involved could take a well-earned break. Countless “third sector” professionals – data analysts, methodologists and administrators – had poured hours of work into their HEI’s REF submission and those who found themselves with some of the heaviest workloads were in the field of impact.
Impact professionals – facilitators, officers and managers – were responsible for the 25% of their HEI’s submission that involved impact. Their critical roles covered the coal-face work of gathering evidence and writing impact case studies, organising internal and external reviews, deciding which cases to submit, providing guidance to PIs and ensuring that case studies were compliant with the guidance. Most importantly, their work helped bring to light the profound and varied benefits that UK research has achieved.
This work was not without cost to these professionals. Dr Julie Bayley describes their experiences and how these were largely negative, for example feeling burned out and pressured by substantial time demands, often while on precarious employment contracts. Now that the REF dust is settling, this community can begin to separate themselves from the general sense of exhaustion and think about why it was such an onerous process.
Two specific areas which made those professionals’ jobs hard was the use of the word “indicative” in the REF2021 guidance and the definition of “formal partner” in the additional contextual information. The overall effect was to take the professionals’ attention away from telling the impact story, as their focus was diverted into trying to interpret imprecise rules, and increased their stress that breaking one of these rules could invalidate the case study.
The relationship between the impact professional and the REF guidance was reminiscent of going to dinner with your other half who can’t make up their mind where they want to go and places the burden on you: “no, you choose”, “where do you want to go?”, “no, honest I don’t mind”.
The overall effect was to take the professionals’ attention away from telling the impact story, as their focus was diverted into trying to interpret imprecise rules
Rather than set specific hard limits for word counts in each section, the number of underpinning research papers and pieces of evidence, REF provided “indicative” maxima for each. This word was the topic of many debates on the ARMA Special Interest Group message boards. Questions such as: “Was it ok to include eight papers as underpinning research?”, “How about twelve pieces of evidence?”, “What if they are labelled “1a, 1b, 1c”? Is that one or three?” obliged these professionals to consult their encyclopaedic knowledge and memory of detailed guidance, often scattered across several documents and FAQ webpages, to find an acceptable answer.
Resolving these issues could have a knock-on effect on the cases. By not including underpinning paper 7, the relevant paragraph would have to be re-written. So do we include it and hope that the REF reviewers don’t notice or mind, or take it out and face another round of re-writing, another few days of waiting on the PI to confirm they agreed to the change, another comment from a reviewer.
Enquiries with REF on what constitutes a “formal partner” created further discussion. The broad interpretation of this term meant that HEIs may define it as every external partner in an EU Consortium, while others might count internal technicians and methodologists and some may leave this section entirely blank. By not specifying what they meant by a “formal partner”, the results may be so varied that they are of limited value.
In contrast, some rules were overly prescriptive, for example around the formatting of numbers. We understand that this was to facilitate data mining for future analysis, but intelligent use of text analysis software at a central level can understand both “11” and “eleven”. Solving this issue would not only free up the impact professionals’ attention for other aspects of the case, but would also provide data that are more meaningful for a national review.
These are two isolated examples of the challenges faced by impact professionals in the preparation of the REF2021 Impact Case Studies. Post-REF decompression meetings hosted by the University of Reading’s Impact Team highlight that many more challenges were experienced, but despite this the impact professionals took enormous pride in the work submitted and wrote the impact cases to do justice to the work carried out by our universities. These people will carry this on from the exclusive focus of the few REF2021 cases studies to the support and nurturing of the many emerging stories in the next phase of their work.
But for the future, a more consistent approach to details in REF exercises would be very helpful: please pick where you want to go for dinner and send us the menu in advance.
Note: This article gives the views of the author, and not the position of the Impact of Social Science blog, nor of the London School of Economics. Please review our Comments Policy if you have any concerns on posting a comment below.
Image Credit: Adapted from Daniel Bradley via Unsplash.