Much of academia has become increasingly influenced by metrics and a set of metrical practices. However, few have totally understood the massive wave of conflicting rules and guidelines that are necessary in order to stabilise these metrical practices. Peter Dahler-Larsen, using examples from his own experiences in Denmark, explains how these multiple, cross-cutting rules have created a disturbing ambiguity in academia. The bibliometric system and the rules which accompany it have created an environment in which many if not most researchers can be identified as transgressors.
This is a new post in the Accelerated Academy series.
Contradiction, paradox, and ambiguity have become ever more apparent in academia in the present era of metrics (including bibliometrics). We need to push our thinking further to understand how that happens. It is common to observe that bibliometrics have unintended effects. (Such as a focus on short-term publication goals, “safe” topics, “slicing” of topics into their smallest publishable units, a “productivity spirit” consistent with “academic capitalism”, etc.). However, to call something unintended is to underestimate its sociopolitical significance.
Instead, I propose to talk about the “constitutive effects” of performance indicators. This concept suggests that our practices are formed, shaped, and reconstituted in the light of indicators which in fact define that which they claim to measure. One of the constitutive effects of indicators is to create more competitive relationships between all those whose scores are compared.
Among other things, constitutive effects are produced through the rules which describe how things should be counted. Some form of bureaucratic control with the categorisation of things is necessary in order to make “systematic” comparison. Furthermore, if people engage in gaming to achieve particular scores, new rules are needed to prevent them from doing so (which were less needed when there was less focus on measurement). Sometimes there are multiple, cross-cutting rules. Together they produce ambiguity. Perhaps this very ambiguity is an important constitutive effect in itself.
Image credit: Background: Rulers 1 by Stew Dean. This work is licensed under a CC BY 2.0 license.
Let me give some examples. The other day I was contacted by my university management who asked me to reconsider how I have registered some publications in the Danish bibliometric database. Bibliometrics influence funding allocations to universities. There was concern in the Ministry of Higher Education and Science that too many publications, in particular chapters in Danish books, were incorrectly registered as “research” where they should have been categorised as “dissemination” (which includes textbooks, etc.). At the same time, they sought to “clarify” some criteria for the categorisation. For example “research” is always peer-reviewed and includes original work (which “dissemination” does not). I argued that the texts in question were surely peer-reviewed and that “results” of research do not rest with empirical data alone. New thinking can be a “result” of research. I further argued that the categorisation itself is problematic. Just because something is dissemination you cannot conclude that it is not research.
Nevertheless, I moved a publication from one category to the other. A couple of days later I contacted the editor of the anthology in which my chapter appeared. He is employed at another university. His own chapters in the anthology had been registered as “research.” I went back to my management with this observation. They responded that the categorisation was really up to me after all. But why did they call the previous registration “incorrect”?
Another example has to do with the recently published Danish Code of Conduct for Research Integrity. This warns against publishing similar results more than once. This should only happen “under special circumstances” and should always be followed by a note explaining where the text was published earlier. The rule was installed after a debate about cases of “self-plagiarism”. (The PhD dissertation written by the minister of research himself was a high-profile example of this debate.) The rule also prevents researchers from earning several bibliometric points from various versions of the same text.
The warning against double publication, despite its apparent logical appeal, has peculiar consequences. In my field it has been common practice to publish similar texts in Danish and English in order to serve both national and international audiences. In compliance with this rule, however, it would be necessary to declare that a text has been published in Danish if it is subsequently published in English. Some editors would probably not like to publicly state that they publish unoriginal material, and so the researcher would lose valuable international bibliometric points. In other words, there would be a strong incentive to publish in English first. And if the text is later published in Danish, it would be necessary to explain that it is a translated English text. (And there would be no bibliometric points from the Danish text, of course). The rules in practice restrict publications in Danish. And why should you have to wait until the English publication is out if there is an ongoing Danish debate which makes the text very relevant here and now? Furthermore, which audience in Denmark would find it necessary to know that this text is a translation of an English one (particularly if it was really written in Danish first?) No one! The purpose of the declaration of original publication is only and exclusively to serve the “publication police” which considers compliance with rules as a purpose in itself. It is not in the interest of the readers. So, should we serve our readers or the publication police?
If we choose to serve our readers in multiple audiences we may be accused of “cheating” in order to earn “easy” publication points. But we carried out the same practices many years before the bibliometric system was installed and so it is clearly not a practice invented to game the system. Instead, it is the new bibliometric regime which now depicts our conventional practice as suspicious.
As the examples show, in a new world full of ambiguity it is difficult to do everything correctly. Either I do not correct the “mistakes” pointed out by the ministry or I register “inconsistently” as compared with my colleagues at other universities. If you look closely enough, the bibliometric system and the rules which accompany it have created an environment in which many if not most researchers can be identified as transgressors.
The Danish code for integrity in research makes secondary publication suspicious. But what exactly is the legal status of that code? It makes reference to the Vancouver protocol. But the Vancouver protocol does in fact admit secondary publication and also specifically mentions the two-language problem. But the Vancouver protocol is also not a legal document. And it clearly states that it is relevant for biomedical journals, but that is not my field. Notice all these but…but…buts. We are in a maze of non-binding and overlapping “rules”.
If the ambiguity to which my examples testify becomes the new normal, any researcher can be penalised more or less randomly for personal or political reasons which have little to do with research quality. We are no longer in a situation where there is a small and clear set of legal rules and the rest is up to professional judgment. Professional judgment is now entangled in a set of metrical practices. And few have totally understood the massive wave of conflicting rules and guidelines that are necessary in order to stabilise these metrical practices. The resulting ambiguity is a disturbing constitutive effect of this whole new configuration of metrics and rules.
Note: This article gives the views of the author, and not the position of the LSE Impact Blog, nor of the London School of Economics. Please review our comments policy if you have any concerns on posting a comment below.
About the author
Peter Dahler-Larsen is a professor at the Department of Political Science, University of Copenhagen, where he is the leader of CREME (Center for Research on Evaluation, Measurement and Effects). He is the author of The Evaluation Society (Stanford University Press, 2012).
I think you confuse “Bibliometric system” with research evaluation. Your problems are related to way in which your government (Research police) is using bibliometrics for your assessment, not of the bibliometrics itself.