As outlined in the Leiden Manifesto, if impact is understood in terms of citations to international publications, a bias is created against research which is regionally focused and engaged with local society problems. This is particularly critical for researchers working in contexts with languages other than English. Peter Dahler-Larsen has developed the PLOTE index, a new indicator which hopes to capture variations in the ability of researchers to spread their impact beyond the English language area.
Everybody is now so interested in the impact of research that the underlying dilemmas and conflicts between various forms of impact are ignored. As the Leiden Manifesto states, if impact is understood as the number of citations of international publications in Web of Science, a bias is created against research which is regionally and locally engaged in societal problem-solving. This dilemma, which is often not debated at all, may be particularly pertinent in the social sciences and humanities and even more critical for researchers working in contexts with languages other than English (LOTE).
There are compelling reasons to publish in English. International journals have high reputations, sophisticated referees, and wider circulation. Nevertheless, there are respectable forms of impact taking place in LOTE contexts. Researchers engage in influencing policymakers, evaluating policies, making policy recommendations, informing local debates, building capacities in local governments and NGOs, educating citizens in scientific thinking, and maintaining a local language capable of dealing with complex societal issues. I asked myself whether a new indicator could be constructed which displays variations in the impact of research across language areas. How do researchers spread their impact in English and LOTE?
I have invented a simple index: the PLOTE index. It measures the percentage of citations which flow from publications in LOTE as compared to citations of all publications. Empirically, most researchers working in LOTE contexts will score somewhere between 0 and 100; the higher the score, the more his or her citations flow from publications in LOTE. In practice, variations in PLOTE reveal variations in the ability of researchers to spread their impact beyond the English language area (the one usually taken for granted).
To show you the practical relevance of these variations I have calculated the PLOTE index for 40 professors in political science in Denmark. In Figure 1, you can see their PLOTE scores plotted against their total number of Google Scholar citations. The trade-off between the two is easy to see, as most researchers are found near to the base of one of the axes. Most researchers who strive for high numbers of citations crawl along the horizontal axis. This is not difficult to understand. (A publication in English gets 18 citations on average, whereas one in LOTE gets only two.) However, we do not know whether publications in English are cited more because of their quality or because of their wider circulation. Maybe both.
Figure 1: Citations and PLOTE-index. This figure is taken from the author’s paper, “Making citations of publications in languages other than English visible: On the feasibility of a PLOTE-index”, published in Research Evaluation (2018).
Most of the researchers with high PLOTE indexes do not have many citations. If you look carefully, however, you will find a handful with high PLOTE scores (around 40) who still have a total number of citations in the 2,000 to 5,000 range. The conclusion is that the road to international recognition typically goes through a low PLOTE score, while a small handful of researchers try to be successful in both dimensions.
At the same time, alongside the strong bibliometric focus on international publication is a contemporary research policy agenda which emphasises the importance of interaction with local and national stakeholders, collaborative research, participation in public affairs, and so on. In the light of this latter concern, a high PLOTE index may be a result of being responsive to this agenda.
The dilemma becomes intensified if you look at PLOTE scores in subfields of political science (Table 1). In subfields which are already thematically international (such as international relations and EU studies) PLOTE is expectedly low (3-5 %). In contrast, researchers in public administration have an average PLOTE index of 32, almost ten times higher than the previously mentioned groups. This is not surprising, as public administration researchers write textbooks, evaluation reports and grey papers in LOTE, as well as manuals and guidebooks for public organisations. A division of labour among subdisciplines is not in itself a problem. However, if research evaluation, recruitment decisions, and departmental strategies are based on conventional bibliometric measures, life will be systematically more difficult for researchers in subdisciplines with high PLOTE scores. Unless, of course, they seek a more international profile, perhaps at the expense of their PLOTE score.
|Subfield||No. of professors||Average PLOTE|
|Political behaviour and institutions||6||11.29|
|Welfare and labour market studies||5||15.21|
Table 1: Average PLOTE-index in subfields of political science among 39 professors. Note: includes professors who could be grouped into subfields with more than four members. One could not be placed in such a group. Data taken from the author’s paper, “Making citations of publications in languages other than English visible: On the feasibility of a PLOTE-index”, published in Research Evaluation (2018).
Faced with such genuine dilemmas, I do not think there is any particular magic number which represents the perfect PLOTE score. However, if PLOTE is measured, and variations shown, the dilemmas for researchers working in LOTE contexts can be appreciated, and be taken into account in research evaluation, recruitment, and strategy-making.
How PLOTE is calculated
To calculate actual PLOTE scores, I have used Google Scholar. A fair objection is that Google Scholar does not measure all impact, only some citations of publications. That is true. Nevertheless, I often find that my publications in LOTE (in my case, in Danish) are vehicles for my attempts to influence a debate, evaluate public initiatives, build capacity, educate citizens, etc. If these publications are cited, at least they are noticed. Google Scholar captures a much broader range of citations than conventional bibliometric machineries. Therefore, Google Scholar also does a relatively better job in describing citations in several language areas. Google Scholar sacrifices validity and reliability in order to achieve comprehensiveness, but it works well for the PLOTE-index, since I am not interested in the precision of any given number, but in showing variations which would otherwise remain invisible.
This blog post is partly based on the author’s article, “Making citations of publications in languages other than English visible: On the feasibility of a PLOTE-index”, published in Research Evaluation (DOI: 10.1093/reseval/rvy010).
Featured image credit: Stanford’s library map of the world on Mercator’s projection by the Norman B. Leventhal Map Center (licensed under a CC BY 2.0 license).
Note: This article gives the views of the author, and not the position of the LSE Impact Blog, nor of the London School of Economics. Please review our comments policy if you have any concerns on posting a comment below.
About the author
Peter Dahler-Larsen is a professor at the Department of Political Science, University of Copenhagen, where he is the leader of CREME (Center for Research on Evaluation, Measurement and Effects). He is the author of The Evaluation Society (Stanford University Press, 2012).