LSE - Small Logo
LSE - Small Logo

Blog Admin

October 1st, 2012

Will the REF disadvantage interdisciplinary research? The inadvertent effects of journal rankings

4 comments | 2 shares

Estimated reading time: 5 minutes

Blog Admin

October 1st, 2012

Will the REF disadvantage interdisciplinary research? The inadvertent effects of journal rankings

4 comments | 2 shares

Estimated reading time: 5 minutes

A failure to engage in interdisciplinary work risks creating intellectual inbreeding and could push research away from socially complex issues. Ismael Rafols asks why there is a bias against interdisciplinary research, and why the REF will work to suppress an otherwise useful body of research.

Since the introduction of the Research Assessment Exercise (RAE) in the late 1980s (now the Research Excellence Framework), there has been a recurrent discussion on whether the RAE would disadvantge interdisciplinary research. In the case of the RAE, neither existence or lack of bias has been unambigously proven. On the one hand, there is a strong consensus from qualitative studies showing that peer review panels such as the RAE/REF have a strong tendency to favour interdisciplinary research. On the other hand, many high profile academics argue that since “interdisciplinary research often lacks rigour”, it turns into a “safe-haven for the lesser-talented”, and it is thus deservedly punished in evaluations . In other words, some believe that evaluations such as the RAE/REF do not treat  interdisciplinary research well, but this reflects the low quality of interdisciplinary departments.

In an article just published in Research Policy, we explore the disadvantage for interdisciplinarity that may result from the use (direct or indirect) of journal rankings in evaluation. Although the RAE/REF is based on peer review, previous evaluations have shown that there is a strong correlation between the journal ranks of the publications submitted and the RAE outcomes of a deparment. As a result, many school managers in areas such as economics and business have been strongly encouraging lecturers to publish in the ‘top’ journals of institutional journal ranking lists. The difficulty for researchers like myself,  working on practice or policy-oriented fields, is that very few ‘top’ journals accept heterodox, interdisciplinary articles. Could it be that there is a systematic bias against interdisciplinarity in the journals rankings?

In order to investigate this question, we first examine the disciplinary diversity of each journal rank of the Association of Bussiness Schools (ABS). It turns out that lower-rank journals (e.g. ‘Acceptable standard’) span evenly a large variety of scientific fields, whereas the top-rank journals (e.g. ´World Elite´) only cover a few disciplines, namely business, management, finance and economics. Hence, according to the ABS ranking, only these four disciplines produce the ‘best’ research. Publications in prestigious journals in other fields are not regarded as ‘good’ research. Given this ideosyncratic perception of quality, it seems a likely possibility that researchers adapt their publications towards ‘better performing’ but a disciplinary less diverse set of journals.

Second, we investigate if the use of the journal ranking would indeed result in an assessment of higher performance by more disciplinary units (in this case, business schools) compared to more interdisciplinary units (in this case, innovation studies units). The challenge for producing this evidence is that both interdisciplinarity and research performance are complex, controversial concepts that cannot be readily apprenhended with uni-dimensional indicators. To address this challenge, we characterize interdisciplinary performance with various contrasting bibliometric indicators. It is found that research in Innovation Studies is consistently more interdisciplinary than business schools across various indicators. However, major contrasts are found between different conventional indicadors when it comes performance. According to performance as inferred from journal rankings, business schools do significantly better than Innovation Studies units. But in terms of number of citations per paper (under various field normalisations), some Innovation Studies units perform as well (or as badly) as business schools. Given that citations per paper is widely acknowledged as a more reliable indicator of scientific impact than journal-based measures, we conclude that the use of journal rankings for performance assessment would lead to a bias against interdisciplinary research. Our findings hold for the field of business and management studies, but can be expected to apply in a other social sciences that use journal rankings such as economics.

The relevance of the study stems from the consequences of the bias against interdisciplinarity in the contents and the societal impact of research. This bias may reinforce disincentives for researchers to engage in interdisciplinary research –e.g. interdisciplinary researchers in some fields were found to be less paid and less promoted; it may create intellectual inbreeding, by which parts of business research become an irrelevant and self-referential game; it may shift research away from socially complex issues, such as climate change; and it may reduce the cognitive diversity of the entire science system, which is important for long term creativity. Although journals rankings are not automatically used in the REF, it is easy to assume that university managers and panel members will be affected by quality perceptions, given the pervasive presence of the journals rankings in business schools.

In summary, the results of our study support what is now a fairly well-established picture from research evaluation studies that criteria of excellence in academia are essentially based on disciplinary standards. If journal rankings are used, consciously or not, to help determine the allocation of esteem and resources in exercises such as the REF, they can supress forms of interdisciplinarity that are otherwise acknolewdged to be academicaly and socially useful.

Note:  This article gives the views of the author, and not the position of the Impact of Social Sciences blog, nor of the London School of Economics

Reference
Rafols, I., Leydesdorff, L. O’Hare, A., Nightingale, P. and Stirling, A. (2012) How journal rankings can suppress interdisciplinarity. A comparison of innovation studies and business & management.Research Policy. Preprint. Interactive Maps.

About the author:
Ismael Rafols is a fellow at Ingenio (CSIC-UPV), Universitat Politècnica de València and SPRU, Science and Technology Policy Research, University of Sussex. He works on the mapping, evaluation and foresight of interdisciplinary fields and emergent technologies.

Print Friendly, PDF & Email

About the author

Blog Admin

Posted In: Impact | REF2014

4 Comments