The Research Excellence Framework (REF) is underpinned by three areas of assessment: outputs, impact and environment. However, discussing the findings of their recent research Mehmet Pinar and Tim Horne argue that these elements correlate to the extent that assessing all of them is largely inefficient. If this is the case, they pose the question: is it time to eliminate one of these elements in the next REF cycle?

The  research excellence framework have just been released. The recent REF is another iteration of the previously employed performance-based research funding systems in the UK, which evaluates an HEI’s research environment, research outputs and the impact of its research. Such an evaluation process is carried out to provide accountability for public money, provide benchmarking information and establish reputational yardsticks, and inform the selective allocation of research funding.

Since its introduction in its current form in 2014, REF has received many criticisms. For instance, the introduction of the impact element to the REF caused some concern among academics as they thought they would be forced to produce more ‘impactful’ research, rather than carrying out their own research agenda.

In short, the evaluation of every element – research outputs, research environment, research impact – may have suffered from bias or subjectivity

Research has found that the evaluation process was biased towards research-intensive universities: Russell Group universities, or universities with panel members involved in the evaluation, tended to obtain unexpectedly higher scores. Even the panellists who evaluated the research elements had problems in doing so. In short, the evaluation of every element – research outputs, research environment, research impact – may have suffered from bias or subjectivity.

Furthermore, REF is expensive, with the cost of carrying out the REF exercise in 2014 estimated to be £246 million.

Given the twin concerns over objectivity and cost, in a recent paper, we use statistical tools to evaluate REF’s usefulness as an evaluation that considers multitude dimensions. If all dimensions are positively and highly correlated, one can obtain similar funding allocation and performance outcomes by using fewer dimensions, suggesting that some of the additional elements are redundant.

Our findings are as follows: the three elements of the evaluation—environment, impact and output—are highly and positively correlated. Therefore, removing a component from the evaluation leads to relatively small shifts in the allocation of funds and in the rankings. As a result, we argue that the REF could be simplified by removing an element, which would reduce the cost of carrying out the REF exercise and result in roughly similar funding allocations and rankings.

This highlights that the total amount of quality-related (QR) research funding reallocated across different HEIs would be just 1.46%, 1.05%, 4.41%, respectively of the amount distributed, when the environment, impact and output element is removed from the evaluation. Clearly these reallocation percentages are small, and so a simplified REF would have little effect on the QR funding distribution, but could reduce the cost of carrying out the REF exercise, and the stress on academic and professional service staff alike.

Based on the findings, two questions need to be answered for the next REF exercise: 1) If funding agencies were to exclude one element, which of the three elements should be removed? 2) If funding agencies plan to include new elements to the assessment, what type of assessment criteria may be included?

For the latter question, our paper provides some hints. Based on our analysis, we argue that if the future frameworks include new elements, these new elements should not be highly correlated with the other existing elements, as otherwise their inclusion to the assessment would lead to increased costs of the assessment, but would produce roughly similar funding allocation and rankings.

On the other hand, our paper does not provide a specific answer to the former question. Yet, we may articulate the potential consequences of removing different elements.

In another recent paper by Pinar and Unlu, they find that the inclusion of the impact element to the REF in 2014 and its increased importance in REF2021, is likely to increase the quality-related funding gap (just over £1 billion mainstream QR funding is distributed every year based on the REF results) among HEIs and in most of the subject areas. Therefore, excluding the impact element from the REF would be likely to result in relatively more equal funding allocations among the HEIs. This is because there is a large gap between research-intensive universities and others regarding the impact element performance.

Alternatively, the performance gap between HEIs is relatively lower based on research outputs, and removal of research outputs is likely to increase the gap between HEIs.

When considering the research environment, a number of papers indicate that quantitative measures e.g., research income, PhD completions, explain much of the variation in scores between submissions; and others indicate that the assessment of the narrative research environment statements may to a significant extent reflect the quality of writing, rather than the underlying research environments themselves. Therefore, a simplified environment element, comprising just metrics, might merit consideration.

Overall, as with any public investment decision, the decision is down to an equity-efficiency trade-off. The efficiency theory argues that the funding should be allocated more towards the most-efficient producers (i.e., better performing institutes). On the other hand, the equity argument suggests that the funding should be distributed more fairly across institutes.

Based on the funding formula used by the funding bodies, the funding bodies clearly favour efficiency, as funding allocated to the world-leading research is fourfold of the funding distributed to the internationally-excellent research, and no funding is provided to the internationally-recognized research. However, there are also relevant arguments for allocating more funding to so-called underperforming HEIs: HEIs that are working towards becoming a more research-oriented university should be protected. This is closely associated with the idea of protecting infant industries.

The UK’s four research funding bodies (Research England, the Higher Education Funding Council for Wales, the Scottish Funding Council, and the Department for the Economy of Northern Ireland) commissioned RAND Europe to carry out real-time review of the REF2021 and offers an opportunity for the stakeholder to give their views on the next REF. Based on our findings, we argue that the basic question of the exclusion or inclusion of assessment criteria should be carefully returned to as part of this review. Whilst the elements of the REF have come to be seen as inherent to the process, perhaps we would be better off without all of them.

This post draws on the authors’ paper, Assessing research excellence: Evaluating the Research Excellence Framework, published in Research Evaluation. 

The content generated on this blog is for information purposes only. This Article gives the views and opinions of the authors and does not reflect the views and opinions of the Impact of Social Science blog (the blog), nor of the London School of Economics and Political Science. Please review our comments policy if you have any concerns on posting a comment below.

Image Credit: j4p4n via open clip art.