Lizzie Gadd argues that any commitment to responsible research assessment as outlined in DORA (Declaration on Research Assessment) and other such manifestos needs to include action on global university rankings. Highlighting four fundamental critiques of the way in which journal metrics and university rankings have been deployed in higher education, she proposes universities could unite around the principle of being ‘much more than their rank’.
More and more institutions are signing up to responsible metrics manifestos such as DORA – which is great. This is no doubt influenced by funder demands that they do so – which is also great. And these manifestos are having a positive impact on researcher-level evaluation – which is triply great. But, as we all know, researcher-level evaluation issues, such as avoiding Journal Impact Factors, are only one element of the sector’s research evaluation problems.
UKRI Chief Executive Ottoline Leyser recently pointed out that any evaluation further up the food-chain in the form of university- or country-level evaluations ultimately has an impact on individual researchers. And of course the most influential of these, at the top of the research evaluation food-chain, are the global university rankings.
So why, I often ask myself, do we laud universities for taking a responsible approach to journal metrics and turn a blind eye to their participation in, and celebration of, the global rankings?
Indeed, when you look at the characteristics of Journal Impact Factors (JIFs) and the characteristics of global university rankings, they both fall foul of exactly the same four critiques.
1. The construction problem
As DORA states, there are significant issues with the calculation of the JIF: the average cites per paper for a journal over two years. Firstly, providing the mean cites-per-paper of a skewed dataset is not statistically sensible. Secondly, whilst the numerator includes all citations to the journal, the denominator excludes ‘non-citable items’ such as editorials and letters – even if they have been cited. Thirdly, the time window of two years is arguably not long enough to capture citation activity in less citation dense fields, as a result you can’t compare a JIF in one field with that from another.
However, global university rankings are subject to even harsher criticisms about their construction. The indicators they use are a poor proxy for the concept they seek to evaluate (the use of staff:student ratios as a proxy for teaching quality for example). The concepts they seek to evaluate are not representative of the work of all universities (societal impacts are not captured at all). The data sources they use are heavily biased towards the global north. They often use sloppy reputation-based opinion polls. And worst of all, they combine indicators together using arbitrary weightings, a slight change in which can have a significant impact on a university’s rank.
2. The validity problem
Construction issues aside, problems with the JIF really began when it was repurposed from an indicator to decide which journals should appear in Garfield’s citation index, to one used by libraries to inform collection development, and then by researchers to choose where to publish and finally by readers (and others) to decide which research was the best for being published there. It had become an invalid proxy for quality, rather than as a means of ensuring the most citations were captured by a citation index.
Whilst the JIF may have inadvertently found itself in this position, some of the global rankings quite deliberately over-state their meaning. Indeed, each of the ‘big three’ global rankings (ARWU, QS and THE WUR) claim to reveal which are the ‘top’ universities (despite using different methods for reaching their different conclusions). However, given the many and varied forms of higher education institutions on the planet, none of these high-profile rankings articulates exactly what their ‘top’ universities are supposed to be top at. The truth is that the ‘top’ universities are mainly top at being old, large, wealthy, English-speaking, research-focussed and based in the global north.
3. The application problem
Of course, once we have indicators that are an invalid proxy for the thing they claim to measure (JIFs signifying ’quality’ and rankings signifying ‘excellence’) third parties will make poor use of them for decision-making. Thus, funders and institutions started to judge researchers based on the number of outputs they had in high-JIF journals, as though that somehow reflected on the quality of their research and of them as a researcher.
In a similar way, we know that some of the biggest users of the global university rankings are students seeking to choose where to study (even though no global ranking provides any reliable indication of teaching quality) because who doesn’t want to study at a ‘top’ university? But it’s not just students; institutions and employers are also known to judge applicants based on the rank of their alma mater. Government-funded studentship schemes will also often only support attendance at top 200 institutions.
4. The impact problem
Ultimately, these issues have huge impacts on both individual careers and the scholarly enterprise. The problems associated with the pursuit of publication in high-JIF journals have been well-documented and include higher APC costs, publication delays, publication of only positive findings on hot topics, high retraction rates, and negative impacts on the transition to open research practices.
The problems associated with the pursuit of a high university ranking are less well-documented but are equally, if not more, concerning. At individual level, students can be denied the opportunity to study at their institution of choice and career prospects can be hampered through conscious or unconscious ranking-based bias. At institution level, ranking obsession can lead to draconian hiring, firing and reward practices based on publication indicators. At system level we see increasing numbers of countries investing in ‘world-class university’ initiatives that concentrate resource in a few institutions whilst starving the rest. There is a growing inequity both within and between countries’ higher education offerings that should seriously concern us all.
What to do?
If we agree that global university rankings are an equally problematic form of irresponsible research evaluation as the Journal Impact Factor, we have to ask ourselves why their usage and promotion does not form an explicit requirement of responsible metrics manifestos. An easy answer is that universities are the ’victim’ not the perpetrator of the rankings. However, universities are equally complicit in providing data to, and promoting the outcomes of, global rankings. The real answer is that the rankings are so heavily used by those outside of universities that not to participate would amount to financial and reputational suicide.
rankings are so heavily used by those outside of universities that not to participate would amount to financial and reputational suicide
Despite this, universities do have both the power and the responsibility to take action on global university rankings that would be entirely in keeping with any claim to practice responsible metrics. This could involve:
- Avoiding setting KPIs based on the current composite global university rankings.
- Avoiding promoting a university’s ranking outcome.
- Avoiding legitimising global rankings by hosting, attending, or speaking at, ranking-promoting summits and conferences.
- Rescinding membership of ranking-based ‘clubs’ such as the World 100 Reputation Academy.
- Working together with other global universities to redefine university quality (or more accurately, qualities) and to develop better ways of evaluating these.
I recently argued that university associations might develop a ‘Much more than our rank’ campaign. This would serve all universities equally – from those yet to get a foothold on the current rankings, to those at the top. Every university has more to offer than is currently measured by the global university rankings – something that I’m sure even the ranking agencies would admit. Such declarations would move universities from judged to judge, from competitor to collaborator. It would give them the opportunity to redefine and celebrate the diverse characteristics of a thriving university beyond the rankings’ narrow and substandard notions of ‘excellence’.
The time has come for us to extend our definition of responsible metrics to include action with regards to the global university rankings. I’m not oblivious to the challenges, and I am certainly open to dialogue about what this might look like. But, we shouldn’t continue to turn a blind eye to the poor construction, validity, application and impact of global rankings, whilst claiming to support and practice responsible metrics. We have to start somewhere, and we have to do it together, but we need to be brave enough to engage in this conversation.
The author is very grateful to Stephen Curry for feedback on the first draft of this blogpost.
Note: This review gives the views of the author, and not the position of the LSE Impact Blog, or of the London School of Economics.
Image Credit: Adapted from Jan Antonin Kolar via Unsplash.
I think we are stuck with metrics, whether applied to journals or universities. It’s a feature of the sheer scale of the respective enterprises (publishing and tertiary education). These are so big and complex that you need metrics to get a grasp on them. So we cannot dispense with them. We just have to improve them. And we are dealing with massive “competitive markets” in both cases. It’s inevitable that these metrics will be used to evaluate in such huge, dynamic and complex “markets”. Good luck!
Thanks Peter. To be clear, I have no problem with metrics per se. I have a problem with poorly constructed, invalid, composite metrics that are compiled into a ranking and claim to provide a legitimate view on the quality of entities, which are then mis-used as such, thus driving perverse behaviours. There are valid and helpful uses of metrics (think gender pay gap data), but the most prominent global university rankings don’t fall into this category.
I agree that some of the measure are worthless; for example, those reply on opinions (I have stopped replying to surveys asking me for my opinions because I just think it is so hard to judge and is likely to be heavily blinkered by one’s own, narrow experience). But people do want to know about the research prowess of a university, they do want to know whether students get jobs, they do want to know whether students are satisfied with the standard of teaching, they do want to know whether the ratios of staff and facilities to student numbers is favourable. I think those are all legitimate measures. If they are flawed in your view, and you think it is legitimate for people to have that information, then I would argue that you need to come up with practical, preferable alternatives. In New Zealand we have got to the point where the main selling point tor many universities is not the quality of research, teaching, facilities, staff, job readiness etc. but whether students can have a good time – in other words, social life, partying etc. I think that has its place, but it should not overshadow the more central criteria, and for that we need decent indicators rather than getting rid of indicators altogether!
“At institution level, ranking obsession can lead to draconian hiring, firing and reward practices based on publication indicators.” Absolutely. The disconnect between signing up to the DORA Declaration and promoting world rankings is very visible. Particularly in the middling rankings, where any rise in rank goes straight on the front page of the uni website. While they are retrenching staff, as in the case of my last couple of employers. Those at the top and the bottom of rankings do not need to celebrate them, or actively try to conceal them. My evidence base is having worked at a community college, and redbrick, white tile, large public, sandstone, and private universities….and Oxbridge and LSE.
“The truth is that the ‘top’ universities are mainly top at being old, large, wealthy, English-speaking, research-focussed and based in the global north.” Classic.
Excluding journal-based metrics from research assessments, global university ranking and funder requirements would eliminate the need to publish in prestigious journals, mitigate the effects of predatory publishers, and facilitate free and immediate deposit (self-archiving) of research in institutional OA repositories. Funds could instead be redirected internally to support and develop research-related processes e.g. peer-review, editing, and translations. As an alternative measure of impact, the global university ranking and funder agencies should consider ranking/assessing/reporting on research articles that support the 2030 UN SDGs.
Global university ranking agencies should become signatories of The Declaration on Research Assessment (DORA) https://sfdora.org/