The recent publication of a series of reviews and the early decisions for REF2029, has highlighted an increased focus on research cultures and environments that extend beyond traditional researchers and research publications. James Baker, Lyndsey Ballantyne, Neil Chue Hong, Gemma Derrick, Andy Dixon, Georgina Fletcher, Jude Fransmann, Simon Hettrick, Emma Karoune, Simon Kerridge, Kirsty Pringle, Tony Roche (The Hidden REF committee) argue that submissions should include 5% of non-traditional outputs and present the Hidden REF 5% Manifesto as a starting point to realise this goal.
Three points stood out in the recently announced early decisions for REF2029.
First, the new outputs criterion, now known as ‘Contribution to Knowledge and Understanding’, continues to support a wider range of outputs than are usually submitted to the REF. As the decision document states these can be: “audio and video content, scholarly editions, software, datasets and artefacts.”
Second, departments are now required to submit structured explanatory statements for assessment along with their submissions. This allows them more flexibility to recognise and value how non-traditional outputs contribute to the overall quality of the research and researchers.
Third, in an unprecedented move for global research evaluation, the UK’s research audit exercise will “allow the submission of outputs by any staff member, including non-academic staff”, this breaking of the link between individual staff members and submitted outputs allows departments a freer hand to select different kinds of outputs.
As we wrote for this blog in 2020, “we believe that if we recognise everyone who is vital to research, we will create the right environment in which to advance it”. We therefore welcome these developments, even if rather than being a revolution, they underscore rules that existed for 2014 and 2021 submissions. However, as previous experience shows, simply allowing more varied contributors and outputs, doesn’t necessarily mean they will appear spontaneously or equitably. Welcome to the new aims and objectives of the Hidden REF.
How risky are non-traditional outputs?
One area of concern arising from the recent decisions centres on the downgrading of the value of outputs in the REF from 65% in 2014, to 60% in 2021 and potentially 45% or less in 2029. The argument runs that submitting non-traditional outputs might come at the expense of a more bankable journal publication, or a double-weighted book. Moreover, the inclusion of a work by any member of staff, may be at the expense of career academic researchers, who traditionally score well as part of peer review-led evaluations. The decision of what, and who’s work to include, is therefore subject to strategic decision-making within institutions.
The perceived risk of valuing non-traditional outputs may not be as high as some think. Looking at the REF2014 and REF2021 data, non-traditional outputs represented only 3.0% (2014) and 2.6% (2021) of all outputs submitted. Submitting journal articles and books was preferred in both exercises. Yet, in both the REF2014 and REF2021, non-traditional outputs did just as well as “traditional” outputs in terms of the proportion awarded a 4* rating. In fact, journal articles perform worse in terms of 4* ratings than the average output submission, only marginally better than physical artefacts, and exhibitions and performances, and substantially worse than digital artefacts, and “other”. Looking ahead to REF2029 the inclusion of a larger proportion of these outputs may therefore not be such a large risk.
Fig.1: REF2021 Output Type proportion changes from REF 2021 Submissions
Fig.2: REF2021 Output Quality Profile by Output Type
Learning to evaluate differently
The Hidden REF first ran during 2020 and was focused on celebrating the variety of ways that traditionally ‘non-REFable’ people and outputs could contribute to research environments and to how knowledge is produced. The exercise highlighted a wide range of under-recognised people and research outputs who/that are essential producers of knowledge. It also required developing new approaches to evaluating and thinking about how non-traditional people and outputs collectively contribute to and are integral modern research culture.
The first Hidden REF exercise made significant progress in this area. Similar to the formal REF exercise, our evaluation panels were composed of world-leading subject experts and panellists with expertise in research evaluation processes. Evaluators were able to find cues and characteristics in which to assess these submissions, despite there being no existing formal training, nor having evaluated these types of submissions before. When evaluating the Hidden Role category, one panellist reported that the “[submission has an] emphasis on recognition of the need for networking, bridge-building, and listening; skills that are not often recognised or rewarded in academia”. Finding these new categories and panel methods and articulating them to submitting institutions, is an important part of demystifying the potential risk of non-traditional outputs. This is a subject we will be exploring further as part of the Hidden REF festival in Bristol on Thursday 21 September.
How much should be submitted?
The recent REF decisions present an opportunity to build a more inclusive research culture, but how much non-traditional work should institutions be aiming to include in their submissions? We propose all Higher Education Institutions (HEIs) should work towards ensuring that a minimum of 5% of submissions under the Contribution to Knowledge and Understanding criterion for REF2029, are non-traditional outputs.
We have named this “The Hidden REF 5% Manifesto” and by signing it, HEIs will indicate their commitment to ensure that the variety of research outputs that make their research environment great, will be acknowledged and take one step further towards a wider range of people being fully valued for their contributions.
Fig.3: Percentage non-traditional outputs by HEI aggregate submission size to REF2021
The 5% threshold is not out of the spectrum of possibility, many smaller institutions and fully 29.3% of all HEIs, already met this threshold during the 2021 exercise (Fig.3). What is stopping a larger representation of HEIs to acknowledge this diversity are fears of being uncompetitive, rather than a justification that it goes against the ‘rules’. The continuing under-acknowledgement of non-traditional outputs in REF2029 submissions threatens to undermine the laudable aims of improving research culture across the sector. Let’s make sure these hidden outputs are seen in 2029.
The Hidden REF Festival will be held in Bristol on Thursday 21 September. To be a part of the change, register here and to show your support for the Hidden REF 5% Manifesto, please sign here.
The content generated on this blog is for information purposes only. This Article gives the views and opinions of the authors and does not reflect the views and opinions of the Impact of Social Science blog (the blog), nor of the London School of Economics and Political Science. Please review our comments policy if you have any concerns on posting a comment below.
Image Credit: LSE Impact Blog via Canva.
