The UK’s periodic research assessment exercise has grown larger and more formalised since its first iteration in 1986. Marcelo Marques, Justin J.W. Powell, Mike Zapp and Gert Biesta have examined what effects it has had on the submitting behaviour of institutions, considering the intended and unintended consequences in the field of education research. Findings reveal growing strategic behaviour, including high selectivity of submitted staff, the reinforcement of scientific norms with respect to the format and methodological orientation of submitted research outputs, and an explicit concentration of funding.
As REF 2021 approaches it is imperative to examine the (un)intended consequences of the first and most institutionalised research assessment exercise in the world. We examine the field of educational research, which, with the “impact” weighting rising to 25%, is of particular interest due to its unquestioned relevance. In our recent paper, we shed light on the UK’s research evaluation system (RAE/REF) and its effects on submission behaviour in the Education unit of assessment. Our longitudinal analysis shows how selectivity changes the research output submitted for evaluation, certain scientific norms are reinforced, and funding is concentrated in a way that consistently benefits the top departments in a stratified higher education system.
Firstly, we traced the evolution of the exercise – its rationalisation, formalisation, and standardisation, as well as its transparency. It must be considered a strong research evaluation system and indeed one that has influenced research evaluation in other countries. Secondly, we collected all submissions to the Education unit of assessment (RAE 2001 – 82 submissions; RAE 2008 – 81 submissions; REF 2014 – 76 submissions; plus the numbers of submitted research-contributing staff for the 1992 and 1996 cycles) and analysed submitted staff and outputs as well as external funding from different sources, whether public, private or international. Thirdly we conducted content analyses of the panel reports in Education and reviewed grant reports to ascertain the funding universities received on the basis of the evaluations. Our findings emphasise that, since 1986, the UK’s research assessment exercise has reshaped educational research in myriad ways.
Evaluation selectivity and strategic behaviour
Which departmental members’ research is selected for evaluation? Has this changed over time? Results show continuous decline since RAE 1996 in the proportion of staff members whose research output has been submitted, which reflects the highly strategic behaviour associated with the evaluation system’s cycles (see Figure 1). Additionally, older, more established universities are dominant compared to the new (or post-1992) universities, reflecting different ages and the reasons for engaging in the evaluation; primarily funding and/or reputation. Perhaps the most striking result is the stable stratification among universities: just ten departments of education in the UK contribute nearly half of the total submitted staff whose research in this field has been evaluated (45% in RAE 2001; 49% in RAE 2008; 46% in REF 2014). These results demonstrate strong “reactivity” that we framed as “reverse engineering”, in which, over time, managers and academics learnt how to “play the game” in order to maximise performance and their departmental results in the exercise, with less regard for the individual or disciplinary levels.
Figure 1: Number of departmental staff submitted for evaluation from pre-1992 and post-1992 universities in Education UoA, RAE 1992—REF 2014. Source: authors’ calculations based on project database (RAE 1992 to REF 2014).
Reinforcing scientific norms
The second result relies on the overwhelming dominance of peer-reviewed articles in the total submission of outputs (see Figure 2). Despite the decreasing number of submitted outputs, due to the selectivity discussed above, articles submitted in REF 2014 exceed the number submitted in RAE 2001, while all other formats of output (books, chapters, reports, conferences proceedings) experienced continuous decreases. We believe this to be a largely unintended consequence of the exercise; although there is no official requirement in the research evaluation system that privileges the research article as the preferred format of output, the bulk of all submissions are articles. Another dimension is the methodological orientation, as the Education UoA panel has encouraged the submission of quantitative studies since the RAE 2001 – “there is room for more approaches that use advanced quantitative methodologies”, (RAE, 2001); “more longitudinal data-sets are needed, for example to provide sound evidence of long-term effects of different factors and innovations on educational outcomes”, (RAE, 2009); “compared to 2008, significantly more outputs were submitted based on structured research designs and quantitative data”, (REF, 2015). Investments in specific initiatives to improve the quality of educational research, such as the Teaching and Learning Research Programme, managed by the Economic and Social Research Council, and the more recent state-funded Education Endowment Foundation, further manifest the preference for submission of more quantitative studies, resulting in intended consequences of policy-driven investments and the influence of the evaluation system and its members, especially panel members.
Figure 2: Formats of submitted publications, by type, 2001–2014. Source: Authors’ calculations based on project database (RAE 2001, RAE 2008, REF 2014).
Funding concentration: explicit stratification maintenance
Finally, our results show that only 40% (2002–2008) and 35% (2009–2014) of funding to universities comes from the quality-related research funding (QR), annually distributed according to the evaluated achievement of each department in the RAE/REF. From 2002 to 2008, external sources provided 60% of funding, while in the next period this rose to two-thirds. The field of education is highly dependent on public funding (from Research Councils and the UK Government). Yet the most striking result uncovers the progressive concentration of external funding and QR funding to the benefit of just a few departments. In fact, from RAE 2001 to REF 2014, the results show the progressive concentration of external funding, with inequality expressed here with the Gini coefficient or index (RAE 2001 – 53.04 Gini coefficient; REF 2014 – 65.77 Gini coefficient). The top ten ranked departments in the last three exercises have succeeded in increasing their proportion of external funding (RAE 2001 – 38%; RAE 2008 – 46%; REF 2014 – 53%), a clear case of the “Matthew effect”.
Regarding QR funding, we find considerable disparities between QR funding distribution after REF 2014 (83.6 Gini coefficient; academic year 2015/2016) and the average in RAE 2001 (51.5 Gini coefficient) and RAE 2008 (59.9 Gini coefficient) (see Table 1). The increased stratification in the concentration of external funding among the top departments may be explained by the capacity of these departments to attract the vast majority of funding from different sources. And the selective behaviour of departments to achieve the best results possible also reflects the policy-driven concentration of funding in the highest-rated universities following the 2003 white paper, “The Future of Higher Education”. After that publication, the Higher Education Funding Council for England ceased to fund departments rated below 4* for the academic year 2003/2004, while in the previous exercise those rated with 3* received some funding.
Table 1. Distribution of external and QR funding, UK (2002–2014). Source: Authors’ calculations based on project database (RAE 2001, RAE 2008, REF 2014) and HEFCE, SFC, HEFCW and DEL Recurrent Grants Reports. Click to enlarge.
Thus, our analysis shows that the institutionalisation of RAE/REF and its growing strength, formalisation, and standardisation has led to various intended and unintended consequences. The UK’s research evaluation system, influential elsewhere, has specific consequences, such as publication selectivity and funding concentration, as it shifts and reinforces dynamics in the contemporary higher education landscape. While it is certainly true that the significant changes for REF 2021 (e.g. all staff with significant responsibility for research must be submitted; an average of 2.5 outputs per staff member must be submitted; and the impact weighting is to increase to 25%) may well alter submission behaviour, there are no signs of counteracting the trend of growing concentration of funds that continues to deepen inequalities between departments and the overall stratification of UK higher education.
This blog post is based on the authors’ article, “How does research evaluation impact educational research? Exploring intended and unintended consequences of research assessment in the United Kingdom, 1986–2014”, published in the European Educational Research Journal (DOI: 10.1177/1474904117730159).
Note: This article gives the views of the authors, and not the position of the LSE Impact Blog, nor of the London School of Economics. Please review our comments policy if you have any concerns on posting a comment below.
About the authors
Marcelo Marques is Doctoral Research Fellow in the Institute of Education and Society at the University of Luxembourg. His current research focuses on institutional change in educational research in England and on the processes of Europeanisation of education.
Justin J.W. Powell is Professor of Sociology of Education in the Institute of Education and Society at the University of Luxembourg. His research centers on the comparative institutional analysis of education and science systems.
Mike Zapp has been Postdoctoral Research Fellow in the Institute of Education and Society at the University of Luxembourg since 2014 and is currently Visiting Research Fellow in the Graduate School of Education at Stanford University. He is interested in the international diffusion of institutional models in education and research with a special focus on the role of international organizations.
Gert Biesta is Professor of Education at Brunel University London and NIVOZ Professor for Education at the University of Humanistic Studies, The Netherlands. He has published extensively on the theory and philosophy of education and on educational and social research.