Research assessment exercises, such as the REF ostensibly serve to evaluate research, but they also shape and manage it. Based on a study of REF submissions in the fields of economics, history, business and politics, Engelbert Stockhammer, argues that the REF promotes a narrow vision of economics determined largely by work published in particular journals and calls for a wider distribution of research funding to prevent fields being captured by dominant academic cultures.

The Research Excellence Framework (REF) is the main research assessment exercise for universities in the United Kingdom. It informs university league tables and determines the allocation of close to £1.6 billion of government research funding. As such, it has a profound impact on the research management of universities and in particular their hiring and promotion decisions. While the REF evaluates the quality of research through a peer review process and explicitly rejects the use of journal rankings, universities widely use journal ratings (notably those established in business schools, such as the ABS journal list) or, more informally, consider the reputation of journals when deciding what submit to the REF.

In a recently published paper we analysed the evaluations of REF 2014 for the units Economics, Business, Politics and History. First, we analysed the journals from which articles had been submitted to the Economics and Econometrics (henceforth: Economics) Unit of Assessment (UOA). A small group of journals dominate the outputs submitted to the Economics. Half of the 2388 outputs that were submitted to Economics were from only 19 journals. This constitutes an extraordinary narrowing of what is considered economics research. For Politics half of the 3082 outputs were from 52 journals. That is a wider range of journals, but again universities seem to have taken a very selective approach to what gets submitted. It is hard not to conclude that journal esteem plays a prominent role within universities when preparing submissions to the REF.

without having read a single paper, but only knowing where papers were published, we are able to explain more than half of the variation of the REF valuations

Second, we investigated to what extent journal impact factors (we use the SCImago Journal Rank) predict the REF´s output evaluations. They do. In simple OLS specifications we are able to explain 80% of the variation of the grade point averages (GPAs) across universities for Economics (The REF evaluates individual outputs, but only the evaluation for academic units; the estimation results refers to the GPA for the submitted units). For Politics these simple regressions explain 58%. Less, but in absolute terms, this is very high. Think about it: without having read a single paper, but only knowing where papers were published, we are able to explain more than half of the variation of the REF valuations (and accordingly funding to universities). Thus, while the REF is based on a peer review process and outputs are assessed independently of where they have been published, we find that journal ratings explain the evaluations of REF 2014 to a large extent.

The REF has strengthened the role the leading journals (and the leading publishing houses). Is that a matter for concern? The answer on that is what those journals publish and whether there theoretical or geographical biases. It is well known that in most fields these top journals are American, which is likely to come with its own geographical biases and paradigmatic preferences.

Economics, here, is an extreme, but hardly unique case, in that it has a rather rigid dividing line between mainstream and heterodox economics. Mainstream economics routinely assumes rational behaviour and clearing markets to then analyse the effect of specific market frictions. Non-mainstream approaches that rely on different ontological or methodological premises hardly ever get published in the top journals. That matters. The REF 2014 took place in the aftermath of the global financial crisis. Mainstream economics’ often assume stable and efficient financial markets Bezemer (2011) argues that while several economists outside the mainstream had predicted the crisis, mainstream economists have failed to do so. Colander et al. (2009) diagnose a systemic failure of economics as a discipline. Heterodox economic approaches, which are built on heterogeneous agents, emphasise the need for psychological and sociological foundations of economic behaviour and have long highlighted the possibility of endogenous financial and economic instability. They could make a vital contribution to enriching economic research and making it more realistic. The continued narrowing of economics, which is bolstered by the REF is thus deeply worrying.

Economics, here, is an extreme, but hardly unique case, in that it has a rather rigid dividing line between mainstream and heterodox economics.

Lee (2007) and Lee et al. (2013) have argued that the REF’s predecessor, the Research Assessment Exercise, substantially contributed to the narrowing of the discipline and its research agenda. Our findings are fully consistent with this. In the REF 2014 hardly any papers from heterodox economics journals were submitted to the Economics UOA. In fact there were more submissions from these journals to the Politics and Business UOAs. Overall, the REF in its present form marginalises heterodox economics, pushes it out of the economics discipline and endangers pluralism in economics research.

But this isn’t just a matter of Economics. The REF has been repeatedly criticised for stifling innovation and being detrimental to pluralism in academic research. Peter Higgs, physics Nobel laureate in 2013, believes that he would not get an academic position in today’s competitive university environment, fostered by the REF. Thomas (2011, pp. 9-10) argues that “In the social sciences, [the REF] has discouraged the writing of books, as opposed to specialist articles, and by making peer review the ultimate arbiter it has very probably enshrined orthodoxies and acted as a curb on intellectual risk-taking and innovation.”

So what could be done? Ideally the design of the REF would be reconsidered to ensure pluralism regarding theories and methodology. But that’s a lot easier said than done. In theory the REF does not consider the publication outlet. In practice there is a very tight correlation between journals, REF submission and REF evaluations. Are universities using journal ratings in their selection processes? Are the panel members unconsciously influenced by journal prestige? Or are top journals simply a valid proxy of high quality publications? Rather than trying to address these questions in the REF process, a simpler way would be to broaden the funding. Today’s funding regime is heavily tilted towards funding ‘world leading’ (i.e. 4*) research. The funding ratio for 3* to 4* is 1:4 (there is no funding for 2* research despite it being ‘internationally recognized’). Recall that for the REF’s predecessor, the RAE the funding ratio was 1:3:7 (for 2*:3*:4*). It needs to be recognised that the evaluations of the REF not just assess research, they are also shaping research. It may support excellent research, but it may also entrench current mainstream approaches and reduce diversity.

Note: This review gives the views of the authors, and not the position of the LSE Impact Blog, or of the London School of Economics.

Image Credit: Amirali Mirhashemian via Unsplash.