The role of bibliometrics, such as impact factors and h-indices, in shaping research has been well documented. However, what function do these measures have beyond the institutional contexts in which, for better or worse, they were designed? Commenting on a study of bibliometrics in the Italian press, Eugenio Petrovich, shows how these numbers play important roles in mediating academic research in the public sphere.

Numbers, in the form of metrics, indicators, and rankings, are increasingly colonising the world of higher education and universities. As in any other part of our social world, however, it is naïve to think of numbers in higher education as neutral pieces of evidence, detached from the social life of people and institutions. Rather, numbers are embedded in discourses, ideologies, power dynamics, policies and politics, and so on. Numbers do not travel alone: they are always attached with social representations, which shape what they mean and what consequences they have. As such, they are always in a process of social negotiation.

Recently, researchers have started to explore the contours of the social negotiation of numbers in higher education. In particular, the numbers produced with the techniques of bibliometrics, such as the Journal Impact Factor or the h-index, have been put under the lens to better understand how they influence the behaviour of scientists and scholars and, more deeply, the very production of knowledge. These studies, however, have mainly focused on intra-scientific contexts and practices, implicitly assuming that the social representations of bibliometrics are made up mainly by academic actors and their satellites. For sure, bibliometric concepts occupy a rather specialised niche and debates around them usually involve mainly the insiders. Still, there are cases in which bibliometric numbers percolate outside the academic arena and reach new actors and contexts.

Numbers do not travel alone: they are always attached with social representations, which shape what they mean and what consequences they have.

In a recent paper published in Scientometrics, I investigated how bibliometric indicators are represented in the Italian generalist press, that is on the newspapers that are commonly read in the country and that shape the public opinion on key matters. Italy is an ideal laboratory to observe the social negotiation of these indicators’ meaning for a two-fold reason. On the one hand, Italy’s research evaluation system is heavily based on bibliometrics. The system was introduced in 2010 as part of a vast reform of the country’s university system, which was heavily contested by the Italian academic community, both with demonstrations and articles in newspapers. On the other hand, Italy lacks a strong indigenous community of bibliometrics experts. Despite the centrality of bibliometrics in research evaluation, no scientific community in the country can claim a professional control over the social discourse on bibliometric indicators. These two factors created the conditions for newspapers to become a privileged arena for the negotiation of the meaning of bibliometrics in Italy, involving a wide range of actors, including academics, journalists, policy officers, and politicians.

Using the online archives of four major Italian newspapers, I retrieved a corpus of 583 articles, published between 1990 and 2020, that mentioned the Journal Impact Factor, the h-index, or other bibliometrics-related terms. The articles were annotated to understand, among other things, who intervene in the press, what rhetorical function the indicators play in the articles’ argument, and in what kind of news bibliometrics appears. In this blog post, I will highlight three main results of this analysis, referring the reader to the paper for a complete account.

The first result is that the Impact Factor (IF, in the following) started to appear in the Italian press in news about scandals in competitions for university chairs. In the early 1990s, it become common practice in medicine to calculate an IF-based metric for individual researchers by summing up the IF of the journals in which they published. In this way, candidates rejected in competitions had at their disposal a new, easily interpretable metric to compare their scientific performance with that of the winners and, thus, reclaim justice in front of the public opinion. In this sense, the IF started its career in the Italian press as a “justice device” to promote meritocracy in academic recruitment, as an objective remedy to the persistent clientelism of academic careers in the country. This narrative framework for the IF grew over the years and reached its highest intensity in the years before the implementation of the 2010 university reform, showing how bibliometrics was embedded into a meritocracy-centered narrative frame long before it was officially enrolled in the Italian research evaluation system (Fig.1).

Figure 1. Articles presenting bibliometric indicators as a means to promote meritocracy in the Italian university system. Figure adapted from the published article.

The second finding is that journalists frequently use the IF as a quality seal for science news: the IF is presented as a warrant of scientific reliability for the news reported, without mentioning shortcomings or limitation of the IF itself. During the Covid-19 pandemic, the h-index was even crudely used to assess the scientific standing of experts that advised the government or appeared in the media, with licenses of expertise attributed or discarded based on h-index scores. A journalist, for instance, explained:

In this evaluation system set up by Scopus, a sufficient mediocrity is reached above 50 points, a certain authority above 80, and thus climbing up to excellence. (“The poorest experts in the world: Burioni, Pregliasco and Brusaferro”, Il Tempo, 2 May 2020)

The third interesting result is the role of amateur bibliometrics in the press, that is bibliometrics produced by nonprofessional bibliometricians. The h-index arrived in the Italian press in 2008, just three years after its creation by Jorge Hirsch. The “carrier” of the indicator was a ranking of Italian scientists known as “Top Italian Scientists” (TIS), published online by the association Virtual Italian Academy. The website offered journalists a ranking of individual scientists that nicely complemented rankings of universities that started popping up in the press in the same years. However, it was the result of a private initiative without any institutional support.

These three findings show the variety of functions that bibliometrics plays in the press and the rich interplay between numbers and social representations. Most of all, they show that, if we want to understand the social life of numbers in higher education and academia, we cannot narrow our view to insiders, but we have to consider the wider society as well.

The content generated on this blog is for information purposes only. This Article gives the views and opinions of the authors and does not reflect the views and opinions of the Impact of Social Science blog (the blog), nor of the London School of Economics and Political Science. Please review our comments policy if you have any concerns on posting a comment below.

Image Credit: Juliana Malta, via Unsplash.