The use of citation metrics in academic hiring and promotion decisions was intended as a response to important and legitimate concerns over the meritocracy of recruitment procedures. However, evidence suggests that doing so distorts scientists’ behaviour and increases the risk that these measures become unreliable. Marco Seeber, Mattia Cattaneo, Michele Meoli and Paolo Malighetti investigated the use of citation metrics within the Italian higher education system and found that scientists responded to a new regulation linking career advancement to metric scores by increasingly citing their own work. Policymakers should take note and use bibliometric tools as a means of informing rather than determining evaluations, as advocated by initiatives such as DORA and the Leiden Manifesto.
Decisions on career advancement are among the most sensitive in academia. In Italy, recruitment procedures have been criticised for an alleged lack of meritocracy, high levels of localism, and cases of cronyism and nepotism. Responding to such concerns, in 2010 the Italian government introduced a national habilitation procedure regulating access to professorial positions. Disciplinary committees could not award a habilitation to a candidate seeking promotion to associate and full professorship unless he or she had met at least two of three indicators of research productivity.
In determining this, disciplinary committees could choose one of two approaches. The first, “bibliometric” approach would take into consideration the candidate’s: i) number of publications; ii) number of citations received; and iii) Hirsch index of citations. In the alternative, “non-bibliometric” approach the candidate’s number of citations would not be considered but instead a minimum number of publications was expected. These publications should be in: i) “first-class” journals; ii) “second-class” journals; or iii) books. (Journals judged to be either “first-class” or “second-class” were publicised in a pre-defined list.)
In so doing, the Italian system became a forerunner in the adoption of a nationwide, metrics-based procedure which would directly impact opportunities for career advancement, and hence a valuable case study to explore the effects of different metrics approaches.
We studied the scientific production of 886 scientists from four disciplines from science’s major research areas – namely genetics (natural sciences), psychiatry (medicine), managerial engineering, and applied economics (both social sciences) – within an observation period beginning in 2002 and ending in 2014. We explored the hypothesis that scientists in those sectors adopting a bibliometric approach would increasingly self-cite in an attempt to boost their metrics scores. A full write-up of our study has recently been published in Research Policy.
Impact on self-citing behaviour
The empirical analysis shows no significant increase in self-citations in disciplines that adopted the non-bibliometric approach. Conversely, in those disciplines that did adopt a bibliometric approach the habilitation procedure is responsible for a significant growth in self-citations. This was especially true in the case of the sector in the social sciences (managerial engineering), where in just a few years (2011-2014) self-citations almost tripled for assistant professors (+179%; +0.85 self-citations), and also increased markedly for both associate (+91%, +0.63) and full professors (+43%, +0.55,). Strong growth in self-citations was also observed in psychiatry for assistant professors (+73%, +1.88) and associate professors (+61%, +1.24), and for assistant professors in genetics (+33%, +0.75). In all three of the sectors we studied which adopted a bibliometric approach, self-citations have grown more strongly for scientists who are relatively less cited.
Figure 1: Evolution of the number of self-citations per article for assistant professors
Figure 2: Evolution of the number of self-citations per article for associate professors
Beyond simplistic metrics
The use of metrics is based on the assumption that publications and citations are proxies of a scientist’s contribution to the advancement of science. However, evidence from countries such as the UK, US and China suggests that using quantitative indicators when making decisions on resource allocation or career advancement distorts scientists’ behaviour and increases the risk that these measures become unreliable. We found that scientists would indeed adopt a questionable practice that is beneficial and not sanctioned. Equally, the use of metrics that are not properly designed is likely to lead to the adoption of questionable practices.
The growing availability of data makes it very unlikely that metrics will disappear any time soon. Moreover, it should not be forgotten that the use of metrics was intended as a response to important and legitimate concerns over the meritocracy of recruitment procedures. Hence, policymakers should carefully take into account the strategic nature of scientists and use bibliometric tools as a means of informing rather than determining evaluations, as suggested by initiatives such as Responsible Metrics, DORA and the Leiden Manifesto, for example. In the same Italian system we studied, some disciplinary committees have recently agreed to consider other criteria in their procedures in order to make assessments more robust, including the monitoring of self-citations. In turn, metrics designers should aim to develop indicators that cannot easily be gamed, nor replace the true goal of advancing knowledge (bearing in mind Goodhart’s law: “when a measure becomes a target, it ceases to be a good measure”). Our own recommendations are that self-citations be excluded from total citations counts, and that additional citations received from the same author would have a reduced value, in order to discourage so-called citation rings and counterbalance the current emphasis on exploitation over exploration in the research agendas.
This blog post is based on the authors’ article, “Self-citations as strategic response to the use of metrics for career decisions”, published in Research Policy (DOI: 10.1016/j.respol.2017.12.004).
Featured image credit: Skitterphoto, via Unsplash (licensed under a CC0 1.0 license).
Note: This article gives the views of the authors, and not the position of the LSE Impact Blog, nor of the London School of Economics. Please review our comments policy if you have any concerns on posting a comment below.
About the authors
Marco Seeber is postdoctoral researcher at Ghent University.
Mattia Cattaneo is researcher at the University of Bergamo.
Michele Meoli is associate professor at the University of Bergamo.
Paolo Malighetti is associate professor at the University of Bergamo.
I would argue that self-citations should never be included in a measure of a publication’s usefulness (which is what citations actually indicate). This post simply confirms & strengthens my point — count ONLY the citations from other scientists.
The issue of citation rings is more difficult, but methods should be developed to surface them.
Software is available that removes self-citations. Fairly simple process.