The UK research system has historically been innovative in its approach to measuring and assessing the impacts of academic research. However, the recent development of the Knowledge Exchange Framework (KEF), has elicited scepticism as to how this framework will significantly differ from the impact element of the Research Excellence Framework (REF). In this post Hamish McAlpine and Steven Hill outline the aims and objectives of the KEF and argue that it provides an important means of understanding the wider totality of research impacts taking place in UK universities.
The recent consultation on the metrics element of the Knowledge Exchange Framework (KEF) has triggered a lively debate on the exercise, its purposes and implications. In particular, there has been some discussion about the relationship between the KEF and the Research Excellence Framework (REF), of which the detailed rules for its second iteration in 2021 have been recently finalised. Mark Reed has questioned the relationship between the KEF and the impact element in the REF. Given that the REF, and the funding that follows from REF performance, already reward impact from research, Gemma Derrick has argued that the KEF might not add any value. In this post, we want to explore the similarities and differences between the KEF and REF impact, and argue that the KEF does indeed provide a different, and important, lens through which to view universities.
Perhaps most fundamentally, KEF and REF have very different purposes. The REF, building on a long history of national research evaluation in the UK, has three principal purposes: to provide information for the performance-based allocation of research funding; to provide accountability for public funding for research; and, to give universities benchmarking information about their performance. More recently, other associated purposes have been identified, in terms of providing national-level insights on research performance, influencing research culture and providing performance incentives. Reflecting these purposes, the REF evaluation is based on absolute criteria that reference international standards of research excellence.
The KEF has a different aim and emphasis. Its primary purpose is to provide institutions with information about their own performance in knowledge exchange in order to facilitate improvement. It also aims to provide information for businesses as well as other users of knowledge, to allow them to better access universities. In order to achieve these aims, especially the first, the KEF is framed around the idea of clustering universities with similar research profiles. Rather than making evaluations against international norms, the KEF seeks to describe performance within these clusters.
There are also differences in the mechanisms of change that are captured in the two systems. The ‘R’ in REF means that the impacts reported there need a clear linkage to research. As the REF panels have pointed out in their recently published criteria, this linkage to research does not assume a simplistic, linear relationship. The pathways between research and impact can be complex, iterative and non-linear, and impact might already have arisen before other outputs of research have been made public. But, nonetheless REF impact is specifically related to research.
In contrast, the KEF looks to capture all types of knowledge exchange, such as those linked to teaching, including provision of Continuing Professional Development courses, the creation of graduate start-ups, and the multi-faceted roles of the University in local growth and regeneration as well as impact from research.
There are further differences in the methodology used in the exercise. The KEF is based primarily on quantitative indicators, for example income derived from various types of knowledge exchange and collaborative or contract research. In these cases, the income to the university acts as a low-burden proxy for the impact generated. There are also other measures proposed, such as academic staff time involved in delivery of activities, companies created, and proportions of publications that have non-academic co-authors. It should also be noted that the currently proposed metrics, are either collected already, or based on readily available data (although some types of knowledge exchange will be assessed on a narrative basis). Hence, the approach will require only a limited amount of additional work by universities, which is appropriate, given that KEF will not be linked to funding for 2019/20.
The largely metrics-led approach of KEF contrasts significantly with the peer-reviewed case study approach of the REF. Preparing a submission for REF impact takes effort for institutions. This effort is rewarded by the creation of a rich evidence-based understanding of how key areas of impact are progressing that can be used to maximise the potential benefits of research. The impact case studies also generate new knowledge about the research impact of the HE sector as a whole; the database of case studies produced after REF2014 has provided an immensely useful source of evidence for further analysis, and we plan to publish a similar dataset after the next REF.
Perhaps the biggest difference between KEF and REF impact is a question of focus. REF impact is very much about outcomes, and because the case studies are selected, they represent the best and most exciting examples of impact. Although 7000 case studies were provided for REF 2014, these represent a small proportion of the total impact that our universities have. This set will inevitably also be biased towards big success stories, and will not generally include examples of smaller scale, but no less important, impacts, types of impact that Gunnar Sivertsen and Ingeborg Meijer have termed ‘normal impact’.
The focus of the KEF is therefore the processes of knowledge exchange, across a wide range of activity. Some of those processes will be related to the big success stories that make up REF impact case studies. But others will lead to smaller, more diverse, but no less important impacts. Others still, will be about building the foundations – relationships and networks – on which future potential impacts will be built. Like the famous quote that genius is 1% inspiration and 99% perspiration, KE is often described as a ‘contact sport’ and this is reflected in some proposed KEF metrics, which are arguably ‘trajectory measures’ which don’t in themselves say much about the impacts realised, but provide a low-burden way to show that the University is undertaking the type of activities at a scale that one might reasonably expect to create impact.
Like the REF, the new KEF metrics approach represents a huge opportunity for universities. It is an opportunity to demonstrate to those outside the sector, the excellent performance in knowledge exchange that is happening across all types of universities, and even to encourage new collaborations. It is a chance to really showcase and value the work that underpins impact.
The KEF also presents a chance for universities to learn through benchmarking their performance against other comparable institutions, not least through providing ready-made peer groups in the form of the KEF clusters. With that learning, there is the potential for an even greater contribution from our world-leading university sector.
About the authors
Steven Hill is Director of Research at Research England and was formerly Head of Research Policy at HEFCE, and leads on all aspects of research policy and funding. He is also chair of the REF Steering Group. He tweets at @stevenhill
Hamish McAlpine is Head of KE data and evidence at Research England, he tweets at @hamish_mcalpine
Note: This article gives the views of the author, and not the position of the LSE Impact Blog, nor of the London School of Economics. Please review our comments policy if you have any concerns on posting a comment below.