Do scholars adjust their publication behaviour depending on the criteria used in their evaluation? Maarten van Wesel presents findings showing how the publishing behaviour of scholars changed when evaluation switched from emphasising ‘publish-or-perish’ to impact factors. Whilst this may suggest a shift from quantity to quality, the number of citations a paper receives not only depends on its scholarly value, but also on seemingly superficial characteristics of a paper.

Evaluation criteria for scholars and universities have recently come under critique in the Netherlands by groups such as Science in Transition, various Nieuwe Universiteit movements, and Platform Hervorming Nederlandse Universiteiten. Elsewhere, movements such as those behind the San Francisco Declaration on Research Assessment have focused on the improper use of the Impact Factor. There are many reasons for these critiques, but it is at least in part a reaction to cases of scholarly misconduct, such as the Stapel affaire, and ensuing debates about whether publication pressures are a cause of such misconduct. But how do evaluation criteria influence publication behaviour?

My present study first looks at the literature about the influence of publish-or-perish attitudes on publication behaviour. This literature is quite extensive and focuses partly on dubious behaviour, such as Salami Slicing and gift authorship, which are both linked to the pressure to publish as much as possible. But the focus is shifting, to the number of citations a paper receives and to the Impact Factor of the journals in which they are published. This shift in focus suggests a move from quantity to quality, as surely researchers and their managers would want to emphasise the scholarly quality and value of a paper!

However, from the field of Scientometrics we learn that the number of citations a paper receives not only depends on its scholarly value, but also on seemingly superficial characteristics of a paper (e.g. van Wesel, Wyatt & Haaf 2014). Factors such as total number of words, the number of authors, author prominence, and the number of references all influence subsequent citation. The authors who, deliberately or not, produce papers with the correct mix of these characteristics become more influential than others in the same field, and will secure more prestige and ultimately perhaps more resources. Because of this virtuous circle, the publication style may be imitated. This is not necessarily unethical, but when these superficial factors are purposely tweaked in order to get more citations it becomes trickier. Adding authors who contributed nothing to the text, lengthening a paper without adding any extra, relevant information, adding unnecessary references etc. can all be seen as misconduct.

Image credit: Crossing Paths by Bala Sivakumar (CC BY)

In the sample used for this study, the rise in the number of authors has been especially spectacular for The Lancet and Nature articles (see table). Whilst the mean number of authors does not rise above ten,  the extremes are spectacular: a Nature paper authored by 241 individuals, almost three times as high in The Lancet 697 individuals. These extremes represent the results of extensive collaborations, but does this justify listing all of those individuals as authors?  These extremes, most likely, do not represent an attempt to beat the publishing game, but might represent an attempt to acknowledge the contributions which were too important to merely mention them in the acknowledgements.  This, however, greatly devalues what it means to be an author. Such practices could be problematic if something is wrong (e.g. plagiarism, data fabrication) with the paper. Are all ‘authors’ willing to accept the responsibility which comes with authorship? In relation to this research, these extremes provide clear evidence for the growing practice of listing non-authors as authors, and the acceptance of this practice by leading journals.

This study also observed an increase in both title length and total number of pages for almost all sets in the sample. The abstracts contain more sentences and become harder to read. All of these characteristics have been associated with a greater number of subsequent citations in previous studies. Of course, all these changes could be due to adjustments of editorial policies of the journals involved. It would, however, be a huge coincidence if this happened in multiple journals from different publishers across diverse fields for various characteristics in the same time period.

Table 1: Author Count Across Journals 1960 to 1974 and 1990 to 2004.

Adapted from Wesel, 2015

There are still other characteristics of papers that are known to influence the number of citations a paper receives, which are untested in this research, most notably the number of references and prominence of the authors. It is likely that further scientometric research will reveal further characteristics which influence the scholarly impact of papers but which are not related to their actual content.

Evaluation criteria continue to shift, due to pressure from scholars and from formal government-sponsored evaluations, such as the REF in the UK. The social impact of papers is on the rise, mapped among others by Altmetrics, but also by scholars self-reporting of news appearances and publications in non-scholarly outlets. Knowledge valorisation is also on the rise as a criterion, relating to how the knowledge gained from research be put in practice. And, maybe more important, how can the results be made valuable in economic terms?

These changing evaluation criteria will lead to changes in the behaviour of scholars, as scholars are highly reflexive and responsive to their environments. For instance, the rising importance of Societal Impact will have a positive effect on the willingness of scholars to write blogposts about their research.

For more on this study and analysis, the full Science and Engineering Ethics article can be accessed, online first, here: Wesel, Maarten van (2015) Evaluation by Citation: Trends in Publication Behavior, Evaluation Criteria, and the Strive for High Impact Publications. Science and Engineering Ethics doi:10.1007/s11948-015-9638-0

Note: This article gives the views of the author, and not the position of the Impact of Social Science blog, nor of the London School of Economics. Please review our Comments Policy if you have any concerns on posting a comment below.

About the Author

Maarten van Wesel works at the University Library and the Department of Family Medicine at Maastricht University. He is an external PhD candidate in the field of Sociology of Science and Technology at the Faculty of Arts and Social Sciences where he is especially interested in the influence of Word Processing on Social Scientists.