This Monday marks the end of the open consultation for HEFCE’s Independent Review of the Role of Metrics in Research Assessment. Steve Fuller expands on his submission and also responds to other prominent critiques offered. He argues that academics, especially interdisciplinary scholars, should welcome the opportunity to approach the task of citation differently. Whilst many complain of the high citation rates of bad scholarship, Fuller wonders if this is a problem of research metrics or of the inability to define a coherent ideal of progressive scholarship.
There is nothing especially wrong with using citation metrics to gauge the state of academic research and the relative position of its players. However, the players need to know the rules of the game, if they are to respond intelligently. To be sure, academics already know that citations operate as market signals by which the fate of their pieces are at least partly determined. Post-peer review citation practices routinely testify to this. So it would be a deep mistake to think that academics are allergic to metrics per se. However, in the past, these metrics were self-organizing and locally enforced. I am not the first to observe that over the 20th century, academia has devolved into fiefdoms and warlords, whose existence tracks the proliferation of journals. Thus, bibliometricians have revealed that the spontaneous arrangement of citations in the field of published knowledge is constructed as mildly overlapping archipelagos (‘networks’ is a bit euphemistic in this context). But academics would need to approach the task of citation differently in light of the current HEFCE proposal – and this should be welcomed.
Image credit: Game of Thrones Board Game, François Philipp (Flickr, CC BY)
If academics are indeed preoccupied with turf-marking in the sense so easily revealed by bibliometrics, then it is easy to see how they might be threatened by any global, unifying perspective as suggested by HEFCE, one that would force you to think about who might take your work seriously outside its default sphere of control. Nevertheless, truly interdisciplinary work tends to become more prominent in this global perspective, if you imagine it as gathering a few citations from many fields over a long period rather than many citations from one field over a short period. The matter isn’t usually put this way because ‘peer review’, the procedure by which research is normally judged, is not seen as ‘quantitative’. But in fact it is: It’s just that the number of people who matter and how they’re chosen is more restricted. To take peer review as the norm is ipso facto to mark research as the inherent property of particular fields, rather than as something of potential relevance to the entire academic community, depending on how it responds.
Notice I am not making a larger claim about opening up judgement on academic knowledge claims to the general public. Rather, I am granting the obvious fact that any potentially interesting knowledge claims have cross-disciplinary relevance, in which case the full academic community – not simply specialists in a given field – should be allowed a free say. (Indeed, even now many works in a given field are prominent mainly because of what they say about things outside their nominal field.) To be sure, citations are not perfect indicators of this phenomenon. However, understood as a globally enforced metric, and with the innovations proposed by the ‘Altmetrics’ crowd, they are a step in the right direction. And yes, academic citation should be seen as a form of marketing, in which authors attempt to couple their fate with others who not only support their knowledge claims but also enjoy an authority beyond that fact.
More specifically, academics should be trained to think about their citations as investments under conditions of scarcity – that is, exactly like a capital resource. A step in this direction would be for journals to force authors to fractionate their citations, that is, to assume that each author casts only one vote in the overall pool of citations. At the moment, any author effectively casts any number of votes through citation, which implies inter alia that heavy citers have a disproportionate impact on the overall citation count. This results in an unhealthy epistemic dependency culture replete in ‘dummy cites’ to people ‘one is supposed to cite’, regardless of true relevance. (I originally raised this point in 1997, in my Open University book, Science, chapter 4, as something that a Martian might recommend to correct the injustices in Earth’s knowledge system.)
To be sure, in the end, we will need to admit upfront that the move to citations is a move to integrate a proper conception of markets into the internal dynamics of academic knowledge production. Given proper regulation (which should be an outcome of the HEFCE consultation), marketization is doable in a way that is accountable not only to the academics who buy and sell knowledge claims through the medium of publication but also to those – that is, the state, the general public and (where appropriate) private funders – who routinely provide relatively stable conditions for such knowledge claims to be transacted.
In this regard, I despair at the ‘critique’ launched by the international relations scholars, Sabaratnam and Kirby, which has received the endorsement of the Campaign for Public Universities and the British Sociological Association. The telling moment for me of ‘Why Metrics Can’t Measure Research Quality’ appears in point 7, which takes on the high citation count for Samuel Huntington, the Harvard political scientist (and former Jimmy Carter advisor) who predicted a ‘clash of civilisations’ between Islam and the West in the early 1990s.
The problem with such scholars complaining about the high citation rates accorded to Huntington is that they end up sawing off the limb of the tree on which they sit. Much of their own scholarship is precisely predicated on Huntington being so badly wrong, which the authors themselves admit is a major source of Huntington’s high citation rate. However, if we didn’t take citation counts seriously, then it would not be clear why we would wish to pay attention to Huntington’s critics, who usually end their diatribes against Huntington, say, by identifying one ‘green shoot’ of an alternative interpretation that empirically suits a specific locale or simply gesturing for a more open-minded view of the role of culture/religion in political life – to be sure, just as long as it doesn’t veer into extreme relativism or (horribile dictu) libertarianism. The critics seem to be incapable of either living with Huntington or without him. But this is less a problem of research metrics than of the old academic left’s inability to define a coherent ideal of progressive scholarship, beyond the identification of scapegoats. In that case, the scapegoats deserve all the citations they receive – and the elimination of citation counts is not a solution the academic left’s problems. However, a new positive vision of the future that attracts citations would do the trick.
This is a slightly expanded version of my submission to the HEFCE consultation on the use of metrics as a basis for research evaluation.
Note: This article gives the views of the authors, and not the position of the Impact of Social Science blog, nor of the London School of Economics. Please review our Comments Policy if you have any concerns on posting a comment below.
Steve Fuller is Auguste Comte Chair in Social Epistemology at the University of Warwick. He tweets at @ProfSteveFuller.
I am one of the original authors of the post cited herein, which was about rejecting metrics as an indicator of research ‘quality’, which is one of the key issues raised by the HEFCE consultation. I note that the author of this post speaks past rather than to the key point, eliding these vaguely as ‘the state of academic research’ and the ‘relative standing of its players’.
We worked with the HEFCE definition of quality as entailing ‘originality, significance and rigour’ and discussed why metrics were a problematic proxy for these specific properties. In a follow-up we also dealt with some of the current problems of REF.
To the extent that we have differences with the author of this post, they are profound philosophical differences about the nature and purpose of academic knowledge. This author proposes a view in which academic knowledge is fully marketised and commodified, based on the logic of demand and supply, and of buying and selling:
“we will need to admit upfront that the move to citations is a move to integrate a proper conception of markets into the internal dynamics of academic knowledge production. Given proper regulation (which should be an outcome of the HEFCE consultation), marketization is doable in a way that is accountable not only to the academics who buy and sell knowledge claims through the medium of publication but also to those – that is, the state, the general public and (where appropriate) private funders – who routinely provide relatively stable conditions for such knowledge claims to be transacted.”
If the market is the arbiter of research ‘quality’, then we need a definition of ‘quality’ which basically means ‘that which is demanded by people willing to pay for it’. I do not share this vision of academic research. Rather I think the first priorities are to produce work which is insightful, honest, rigorous and accountable, and to cultivate spaces for independent and critical thought.
Regarding Huntington, it was an extreme example to make a point, but the same point about the ambiguity of citations could be applied equally to Wendt’s 1992 piece ‘Anarchy is what states make of it’.
I am as puzzled by Professor Fuller’s response to our paper as Meera is. The various conflations of content, popularity, readership and quality is unfortunate, given that we make some effort to clarify these distinctions in our submission. (Incidentally, we never once write “critique” in that submission, so Professor Fuller’s use of that term in scare quotes is hard to understand except as a sneer at some straw figure).
Although the example of Huntington is supposed to undermine our argument, it does quite the opposite. HEFCE have asked about the use of citations as measure of quality. Professor Fuller suggests that the many critics of Huntington do not innovate much, and yet their inclusion in the Huntington controversy boosts their profile. This, we would suggest, is not a reason to regard either their work or the original as high quality research. Why, then, would we care about Huntington’s critics? Perhaps on the content of their arguments, rather than on either their citations or the citations of the work they are responding to. As we indicate in the paper, the fact that a piece is highly cited may of course tell us that it has generated a certain level of interest, but the leap from popularity or notoriety to a judgement of research quality cannot be achieved without the circular logic that Professor Fuller indulges, in which something that is highly cited is good on the grounds that it is highly cited.
Professor Fuller seems to think that we not only need that guide to what to read (and I can happily concede that we are often led by the buzz around a given work), but that the citation guide should be institutionalised and become central to the dispersal of research funds. His comments on the training we are all to undergo indicate that he knows exactly the problems that are associated with citations as a measure of much at all, and yet he also recommends that we get better at gaming (sorry, “marketing”) those claims to expand our reputations. This is a recommendation not only that HEFCE adopt metrics in some form, but that they elevate them to the supreme arbiter of academic fates, necessitating a wholesale transformation of academic practice in line with a system of wholly artificial “scarcity”.
This vision does not cohere even if we grant these assumptions. We are told that it is an “obvious fact that any potentially interesting knowledge claims have cross-disciplinary relevance”. This is neither obvious nor a fact. A breakthrough in cancer research would revolutionise oncology, and then the lives of billions, but is unlikely to feature heavily in any International Relations bibliography. The discovery of a new Shakespeare portfolio would transform the teaching of English forever, but will not much bother medical anthropologists, and so on. In some fields (perhaps such as Professor Fuller’s), powerful claims might spill over into other domains. In others they would not. Moreover, because different disciplines have different citation practices, an article in one field might well accumulate more citations within that field than a different piece would across several fields with a lower propensity to cite. In other words, aggregate citations are not even an indicator of cross-disciplinary take-up.
Perhaps that is another area in which we need further disciplining. But as it stands, there remain no good reasons to subsume judgements of quality under a unified hierarchy of accounting.
Thanks for these two responses. I think you’re approaching the matter too narrowly, but maybe my background assumptions are not sufficiently explicit.
Who controls academic quality? Should it be the producers or consumers of academic knowledge? My answer is that it should be those consumers who are also producers, which is best captured through citation metrics. I also believe in a free trade policy across academic fields, in which case the value of the knowledge produced is determined by those who accord it value in their own practice, regardless of field. If we don’t assent to this principle, then I don’t see why we’re talking about ‘academic knowledge’ as such rather than some more general or specific category of knowledge production and consumption. Understood the right way (and this is a big ‘if’, which HEFCE should spend most of its time on), metrics potentially provide a powerful way of democratising academic knowledge production. The real question is how to get both ordinary academics and policy makers to think about metrics appropriately. E.g., the default idea that ‘one cite = one vote’ is clearly wrong.
In this context, journals and other peer-reviewed academic gatekeepers are really no more than filters for defining something qualifying as ‘academic’ knowledge so it can enter the citation market in the first place. After that, these agencies should exert no more control. Journal placement certainly doesn’t determine ‘quality’ in some epistemologically luminous sense. (This is one reason to kill once and for all the ‘journal impact factor’ as a global metric, which merely encourages new authors to cite old authors in the journals where they’d like to appear.) To think that it does is to resort to the worst sort of academic protectionism that deserves the name ‘fiefdom’ and does nothing to improve the flow and character of knowledge production either inside the academy or in its relations to the larger world.
‘Academic quality’ is not a matter of who is right or wrong but who is adding to overall academic value. So if someone like Huntington gets a lot of negative cites, then he has at least helped to orient a larger academic discussion, which is a very worthwhile thing even if his own views are dismissed in the long run. Not every idiot or bigot — or even tenured professor — achieves so much over his/her career! Again, my challenge to those who object to Huntington’s skewed influence is to come up with something better that attracts the same level of cross-disciplinary interest – and citations can measure that.
I don’t deny that academic knowledge production is skewed by the people willing to pay it. On the contrary, I am very much opposed to the way in which academics on big grants are presumed to be contributing to the quality of academic knowledge simply by contributing to their universities’ coffers. My own informal view is that big grant getters are like big gas guzzling cars – totally inefficient from the standpoint of producing academic knowledge. However, my guess is that they’re favoured – beyond their publicity value – because the grant guzzlers enable the university to offload some of its normal labour and resources costs so it is not so heavily dependent on direct subsidy from the state. But this raises a more general issue about how universities are encouraged to fund themselves. However, there is a specific role for HEFCE here in terms of looking across fields and evaluating the basic relationship between grant money and academic output. E.g., is there an optimal grant size?
It may be that in the end the source of our disagreement is that I want to protect the integrity of ‘academic knowledge’ as such rather than the integrity of particular fields of knowledge and the interests currently associated with them. However, I am not oblivious to the larger background issues of certain fields or ideas attracting more money and thereby skewing the overall flow of knowledge production in the sort of free market environment I’m advocating. Nevertheless, if we wish to get a grip on how those external factors skew the internal ones under which we normally work, then we will need some sort of common unit of exchange to identify and assess influence in a relatively exact and controllable way. And the sort of quantification provided by metrics is a welcomed step in this direction, provided that the problem is considered at the comprehensive level that I have been suggesting.
These clarifications are indeed helpful, although the underlying philosophical differences between us remain. Your comments on impact factors are also entirely fair, but reflect rather than refute our position (it is another curiosity of your response that you seem to take us as defenders of the impact factor, which we are not). I would only make two additional points.
First, you are operating with a different understanding of quality to that enshrined by HEFCE (you prefer “adding value”, they prefer “originality, significance and rigour”). Since HEFCE are asking about whether metrics can reflect quality in their sense, and whether it should influence the future dispersal of funds that they oversee, the narrowness of our response seems entirely appropriate. Although you concede both disciplinary differences, and forms of gaming, you suggest that being cited more should nevertheless act as the universal currency of academic status. The reasoning on this remains circular as far as I can make out: you take it for granted that an author adds value if they are highly cited (value being understood as generating many citations) and then respond to the criticism of citations as a metric of value by demanding that critics get themselves highly cited! (I will leave others to ponder the comment that funding on the basis of quality in research – including for the hard sciences – should not be judged on the basis of being right or wrong).
Second, you place great trust in the ability of metric to accomplish the role of rigorous measure of value, or cross-disciplinary interest, or useful framing. We have, I think, provided thus-far unanswered reasons as to why it cannot accomplish this in principle, but others (such as David Colquhuon and David Spiegelhalter) have rather compellingly set out why there are much more specific technical and logistical reasons why such counts would be fundamentally misleading. There is nothing yet to suggest that these suspicions of quantification are misplaced.
I have now read the papers by Colquhuon and Spiegelhalter that you linked, and it’s clear that an intellectual sea change is needed to take metrics seriously.
Both papers highlight obvious problems with interpreting metrics crudely or naively. I couldn’t agree more. What is not so clear is why these are arguments against metrics as such rather than for improving metrics – especially if one considers the alternative suggested by the authors, which is a regression to the sort of provincialized discretion associated with the normal ‘peer review’ process. The one inherent virtue that metrics has over peer review is that, in some sense (albeit imperfect), each knowledge producer counts in the final judgement, whereas peer review is a ‘representational’ process in which certain elite members of a field stand for the entire field – but without the formal consent of the all of the field’s members.
It is thus easy to see how peer review can generate fiefdoms. The tendency is revealed when someone calls for comparisons of what metrics deliver with the ‘consensus of expert judgement’, as if the former need to be held accountable to the latter. Doesn’t this just beg the question of the value of metrics? The test for metrics is not whether they end up reproducing the academic status quo. It is not like an exercise in artificial intelligence, where one might conclude that the computer fails to match human performance. Rather, the drive toward metrics is motivated by a general dissatisfaction with the communication and representation of academic knowledge, given how much of it is being produced and how relatively little of it is being used by either academics themselves or the wider society. In that case, we should be open to the prospect that a sophisticated set of metrics might generate an epistemic horizon interestingly different from the status quo. This is why I refuse to get involved in discussions of ‘right/wrong’, because in practice these tend to be mystified euphemisms for the ‘consensus of expert judgement’. And as for Goodhart’s Law, let’s get real: Even in the ‘qualitative’ world of peer reviews, academic players are always gaming the system by writing and citing in ways that will get them accepted by the favoured journals. Going quantitative will simply expose them to a larger field of play and make the practice explicit and monitorable.