LSE - Small Logo
LSE - Small Logo

Blog Admin

January 26th, 2016

Are scientific findings exaggerated? Study finds steady increase of superlatives in PubMed abstracts.

19 comments | 7 shares

Estimated reading time: 5 minutes

Blog Admin

January 26th, 2016

Are scientific findings exaggerated? Study finds steady increase of superlatives in PubMed abstracts.

19 comments | 7 shares

Estimated reading time: 5 minutes

CC BY Chris PotterAre scientists using language aimed at convincing editors and reviewers to publish their work? Joeri Tijdink, Christiaan Vinkers and Wim Otte present findings which suggest a rise in potentially exaggerated language. Potentially conflicting with the core values of science, the pressure to publish in high impact publications may be contributing to a paradigm of over-interpretation, overstatement and misreporting of scientific results.

Our perception and beliefs of how the world works are influenced by the way we use language. This is no different in science. Scientific language has the potential to influence not only readers, but also editors, reviewers and colleagues and push opinions and thoughts in certain directions. The use of convincing language might consequently be essential to sway readers to foster credibility of scientific work. Language used in scientific articles have the potential to ‘make it or break it’. Consequently, it may well be the case that scientific findings and discoveries are sometimes exaggerated and the implications of the research overstated.

Another potential factor to bear in mind is the rapid and steady growth of the scientific enterprise. Between 1996 and 2011 over 15 million individuals authored around 25 million papers resulting in increased competition for the very few publication slots in high impact journals. These high impact publications are particularly important for the career of a scientist. This may have contributed to a paradigm of over-interpretation, overstatement and misreporting of scientific results in order to get published.

1851-happy-sadImage credit: Prerana Jangam Public Domain PDPics

There is a well-known tendency in people to use positive words, and exaggeration of research related news has previously been linked to overstatements in academic press releases (Sumner 2014). In order to examine the use of superlatives in science, we used a data-driven approach examining trends in the use of positively and negatively valenced words in PubMed abstracts and titles over the past four decades. We hypothesised that the emergence of a ‘publish or perish’ culture aimed at productivity and novelty could have affected the use of positive and negative words in scientific reporting and discussion.

We quantified the yearly frequency of predefined positive, negative, and neutral words (25 words in each category) in titles and abstracts obtained from the PubMed database (box). Analyses were restricted to the period from 1 January 1974 to 31 December 2014, to ensure that all abstract texts were available. Words were selected after a consensus between the authors was reached through discussion, which included manual analysis of random abstracts and search of thesaurus listings. To validate the results from these pragmatic prespecified lists. To further exclude a bias in the choice of these words, we also searched for 50 nouns and 50 adjectives randomly selected from Ogden’s 850 core words of Basic English.

Results

The absolute frequency of positive words increased from 2.0% (1974-80) to 17.5% (2014), a relative increase to 880% over four decades. All 25 individual positive words contributed to the increase, particularly the words “robust,” “novel,” “innovative,” and “unprecedented,” which increased in relative frequency up to 15000%. For full results see here.

F1.largeImage credit: Vinkers et al (2015). Use of positive and negative words in scientific PubMed abstracts between 1974 and 2014: retrospective analysis. BMJ. Creative Commons Attribution Non Commercial (CC BY-NC 3.0) license

The increase in positive word use over the last ten years was significantly lower in high-impact factor journals, indicating that the upbeat trends may be less pronounced in these journals. We thought this may have been caused by a more intense peer review process with more reviewers involved and a stricter and more professional editors board.

Reflections

One important problem science is facing is the ‘publish or perish’ culture. In this culture, high-impact publications pave the way for recognition and the chances for success. Measurement of scientific quality is predominantly built on the number of publications and citations. In this context, it is understandable that scientists want to publish as much as they can in high impact journals. The more novel and groundbreaking your research is, the sooner it will get published.

This, in turn, will increase your chances in grant procedures and foster your career potential (the credibility cycle, see figure below). It is therefore rather logical that scientists use the language that can convince editors and reviewers to publish their work. This can potentially conflict with the core values of science such as objectivity and trustworthiness. Instead, the scientific enterprise should aim at these core values to highlight nuance and objectivity to pursue a genuine quest for truth.

credibility cycleFigure: The credibility cycle, adapted from Latour and Woolgar. Points at which organizational devices connect to the cycle are shown.

For the general public, scientists are thought to know how the world really works. This causes a societal pressure for real scientific progress (e.g. improved treatment of cancer or dementia). As Christiaan Vinkers recently explained to Retraction Watch:

Rather than giving in to this pressure to deliver, we need to be realistic in what science can offer and the speed in which developments take place. This needs not only be communicated to the general public but also to news media. They sometimes unthinkingly write on science results without proper context and without the nuances and limitations that are essential for a correct understanding and interpretation of scientific results. For scientists, some of the more direct possibilities to improve the current situation relate to the current publication culture and the way science quality is (bibliometrically) measured. Progress in science is essential but needs to be based on realism. There should be room for ‘slow science’…

Changing the culture

It is not easy to push a culture towards another direction. Besides, the current academic structure is complex and it is hard to tell which interventions will work and for whom they should apply. Nevertheless, change starts with increased awareness among all parties involved. In this light, the good news is that numerous initiatives across different scientific areas have recently emerged, such as METRICS, the DORA manifesto, Force11, ALTmetrics, Science in Transition, the REWARD alliance) These initiatives are valuable and will eventually result in new values and forms to reshape current publication practices.

Despite the reported steady increase of superlatives in science, this finding should not detract us from the fact we need bright, unique, innovative, creative, and excellent scientists that however do write objective and nuanced scientific articles.

This post is based on a journal article originally published in the BMJ 2015; 351Use of positive and negative words in scientific PubMed abstracts between 1974 and 2014: retrospective analysisdoi: http://dx.doi.org/10.1136/bmj.h6467 (Published 14 December 2015)

Note: This article gives the views of the author(s), and not the position of the LSE Impact blog, nor of the London School of Economics. Please review our Comments Policy if you have any concerns on posting a comment below.

About the Authors:

Joeri Tijdink is a psychiatrist at Waikato hospital in Hamilton, New Zealand. He is currently finishing his PhD at VUmc Amsterdam entitled Publish and Perish; research on research and researchers that aims to uncover the major pitfalls in contemporary publication practices with a focus on the role of individual factors of biomedical scientists (burnout symptomatology, personality traits). He and Christiaan Vinkers are also the co-founders and editor of the Dutch platform for young psychiatrists (dejongepsychiater.nl, @jongepsychiater) that focuses on translation of psychiatric research into clinical practice.

Christiaan Vinkers is a psychiatrist and researcher in the Rudolf Magnus Brain Center. He studied Pharmacy (2005), Law (2009), and Medicine (2009) at Utrecht University and finished his PhD thesis on fundamental neurobiological research and the GABA system in 2009. Subsequently, he started his residencies and became a psychiatrist at the Rudolf Magnus Brain Center in 2014. His overall research objective is to investigate the neurobiological background of stress resilience and vulnerability including (epi)genetic, neuroendocrine, and brain circuitry factors, with a special interest in the GABA system.

Wim Otte is a neuroscientist at the Center of Image Sciences and Department of Child Neurology at the University Medical Center Utrecht. His main research interests include epilepsy, stroke rehabilitation, prediction modelling and global health. He is chair of the non-governmental organization CBR Effata which runs several community-based rehabilitation projects in rural Guinea-Bissau and Nigeria; (it provides him with an excellent excuse to escape the continuous publication pressure and the bad Dutch weather several times a year).

Print Friendly, PDF & Email

About the author

Blog Admin

Posted In: Academic publishing | Evidence-based research

19 Comments