What exactly is a 4 star REF output or impact case study worth? Jon Collett runs the numbers and suggests how universities might use this information to shape submissions for REF 2029.

With the announcements in summer 2023 of the next REF, I asked myself, how much are the REF2021 impact case studies and outputs worth to universities? As any impact case study manager worth their salt would do, I decided to quantify the benefits of my work to the beneficiary! The Fast Track Impact blogpost by Simon Kerridge and Mark Reed, “How much was an impact case study worth in the UK Research Excellence Framework?” provided a fascinating insight into how REF2014 scores directly affected the quality related (QR) funding awarded to institutions. This post discusses my findings, based on the REF2021 data.

REF2021 and QR Funding

REF2021 is simpler than its predecessor to calculate values of 4* impact case studies and outputs. There was no impact statement, meaning that all of the impact score could be directly attributed to the case studies returned. For outputs, the requirement of 2.5 outputs per FTE returned means that we can be confident of how many publications were assessed per institution, as the possibility of a reduction at the submission level was not often used.

The data around QR funding is transparent. UKRI publish the QR allocations to HEIs on their website with granularity down to the proportion each sub-profile of each unit of assessment (UoA). However, it is only the 2023-2024 QR allocations that we are looking at in this analysis, which uses the scores from REF2021. Therefore, this article is only looking at the value of 4* impact case studies and 4* outputs based on QR allocations of a single financial year. (As an aside, QR funding in 2022-2023 also uses the REF2021 scores, and is almost identical to the 2023-2024 data).

Calculating 4* impact case study and 4* output values

The annualised data provided by UKRI is granular and is split into the outputs, impact and environment parts for UoA, and also include data such as the number of FTE returned. As such, combined with the knowledge about number of impact case studies and outputs required per FTE returned, the annual values for 4* outputs and impact case studies for individual submissions of each UoA at each institution can be calculated as follows (where ICS stands for impact case study):

These equations will give the value of a 4* impact case study and a 4* output to that submission from an institution in the 2023-2024 financial year, so long as it received QR funding.

Relative values of 4* impact case studies and 4* outputs

Following the described method, all of the values of 4* impact case studies and 4* outputs in 2023-2024 can be calculated on a per submission and per institution level. As this method can be applied to all institutions in receipt of QR funding, we can visualise the data using a boxplot to demonstrate the distribution and skew of the values of impact case studies per UoA, as seen in Figure 1.

Figure 1: Box-Whisker statistical plot of all 4* impact case study values per UoA in 2023-2024. The mean line was added to improve visibility of the mean average in the plot. Plot omits values with a multiple submission letter for UoAs 3, 22, 25, 26, 28, 32, 33 and 34.

The first observation of the distribution of data is the wide range of values that an ICS can have in a single UoA. To understand why this is the case, the QR funding has been normalised to the 4* impact score, meaning that the value of 4* impact case studies is solely driven by the FTE returned. Therefore higher FTE means each individual impact case study will be worth more, as the number required does not proportionally increase with FTE. As an example, General Engineering (UoA12), had an order of magnitude difference in impact case study value between the institution with the most FTE returned and the one with the least.

Figure 2 further emphasises the relationship between the size of the return by FTE and the 4* impact case study value, and the differing values between panels.

Fig.2: Scatter graph highlighting the relationship between size of submission size return by FTE versus 4* impact case study value in 2023-2024. Plot omits values with a multiple submission letter for UoAs 3, 22, 25, 26, 28, 32, 33 and 34.

As the number of outputs required by REF2021 did increase proportionally with FTE, the value of each one should therefore be relatively consistent across institutions in the same UoA. This can be seen in Figure 3, which also demonstrates the relatively constant value of each 4* output in the same REF panel.

Fig.3: Box-Whisker statistical plot of all 4* output values per UoA in 2023-2024. The mean line was added to improve visibility of the mean average in the plot. Plot omits values with a multiple submission letter for UoAs 3, 22, 25, 26, 28, 32, 33 and 34.

How much is a 4* impact case study worth?

As the values of impact case studies are generally similar for UoAs within the same panel, a good approximation for value will be to look at the median values of each panel, shown in the table below (N.B. median calculated omitted values with a multiple submission letter for UoAs 3, 22, 25, 26, 28, 32, 33 and 34).

From this, a good rule of thumb would be that a 4* impact case study in Panel A and Panel B was worth around £110k in the 2023-2024 financial year, whereas for Panel C and Panel D a good approximation would be £60k over the same period. However, the large spread of values means that even looking at averages would overvalue them to small submissions and undervalue them to large UoAs. This would then have a knock-on effect to the potential strategies adopted by each institution. A far better solution would be for each institution to make their own calculations to what outputs and impact case studies are worth to them, and use the ratio between their values to inform strategy.

How much might a REF2021 impact case study be worth over the whole REF cycle? This is really hard to discuss as we are only in the second financial year of the current cycle, and there are a further six financial years to come before REF2029 results might be used for the QR funding (if this calculation is even still used). Making a (rather large) assumption that the QR funding will remain at 2022-2023 levels for the rest of this REF cycle, we can see that the median ‘rule of thumb’ (based on annual values) for 4* impact case studies will be approximately £880,000 for Panels A and B, and £480,000 for Panels C and D over the REF cycle. However, as there is such a distribution of values depending on FTE of the submission, I encourage each institution who are interested to make their own calculation.

Considering the value of a 4* impact case study over a whole REF cycle, in a 2023 article on Research Professional Simon Kerridge estimated that from REF2029 a single 4* impact case study from a very large UoA could be worth over £2 million. The data presented here shows that over the eight financial years that REF2021 results will likely be used for the QR funding allocation, eight of the largest submissions have already achieved this value.

What else can we do with this information?

First, we should discuss what we should not do with this information: comparisons. Each submission in each institution is unique, and using information like this to compare between them is like comparing apples with pears. Each University has its own circumstances that lead to different considerations and decisions being taken, making comparisons between them is unfair.

So, what should we use this information for?

Even though I have not provided a set value for a 4* impact case study, this analysis can be used to inform strategy development as we approach REF2029. For example, if maximising income is your goal, should your institution focus time on maximising the number of 4* outputs or on developing impact case studies to try and ensure they are 4*? If you know that you are preparing the REF return in a submission with less than 100 FTE, you may choose to focus on maximising the number of 4* outputs you can return and the new impact statement (when we know more details), rather than focus on developing individual 4* impact case studies. If you are supporting a return in a submission with more than 100 FTE, you may choose to focus on maximising the chances that each of your impact case studies are 4*, as they are disproportionately much more valuable than individual outputs. Submissions of less than 40 FTE may even choose another strategy to maximise income.

Another important use I have found with this information is training and communication around REF. Being able to communicate to researchers the value of each impact case study or output, with an explanation of how the UK’s dual research funding system works, helps secure their support and cooperation for REF. I work in an Engineering Faculty so I find this particularly true when I can use numbers and values to explain the context of how their work benefits from the QR funding being awarded to the University.

Discussing impact case study evidencing requirements with other parts of my organisation is also helped with this information. For example, when in discussions with my institution’s Technology Transfer Office around requirements for impact case study evidencing, being able to provide a value around additional income the University could receive through QR funding helps generate an understanding of why I am asking for specific types of evidence for impact.

Final thoughts

This analysis is looking back at how REF2021 scores have influenced QR funding. Preparations are building to REF2029, which will have different rules emphasising inclusion and diversity, and there is no guarantee that QR funding will remain as it is after the results are published. Therefore, maybe the best strategy will be to not worry about the financial benefits of future REF, but focus on how to build the type of research excellence and culture that REF2029 is aiming to measure.

Readers can explore the underlying data for this analysis here. This post originally appeared as, How much are REF2021 4* impact case studies and 4* outputs worth? on the Fast Track Impact Blog.

The content generated on this blog is for information purposes only. This Article gives the views and opinions of the authors and does not reflect the views and opinions of the Impact of Social Science blog (the blog), nor of the London School of Economics and Political Science. Please review our comments policy if you have any concerns on posting a comment below.

Image Credit: wutzkohphoto on Shutterstock.