LSE - Small Logo
LSE - Small Logo

Blog Admin

August 23rd, 2016

Bias against novelty in science: A cautionary tale for users of bibliometric indicators

1 comment

Estimated reading time: 5 minutes

Blog Admin

August 23rd, 2016

Bias against novelty in science: A cautionary tale for users of bibliometric indicators

1 comment

Estimated reading time: 5 minutes

Lego_breakthrough_wallNovel breakthroughs in research can have a dramatic impact on scientific discovery but face some distinct disadvantages in getting wider recognition. Jian Wang, Reinhilde VeugelersPaula Stephan present an overview of their findings which suggest an inherent bias in bibliometric measures against novel research. The bias is of particular concern given the increased reliance funding agencies place on classic bibliometric indicators in making funding and evaluation decisions.

There is growing concern that funding agencies which support scientific research are increasingly risk-averse and that their competitive selection procedures encourage relatively safe and exploitative projects at the expense of novel projects exploring untested approaches. At the same time, funding agencies increasingly rely on bibliometric indicators to aid in funding decisions.

Research underpinning scientific breakthroughs often is driven by taking a novel approach, which has a higher probability of major impact but also faces a higher risk of failure.  It may also take longer for novel research to make a mark, either because of resistance from incumbent scientific paradigms or because it takes longer to incorporate novel contributions into follow-on research.  The “high risk/high gain” nature of novel research makes it particularly appropriate for public support.  If novel research, however, has an impact profile distinct from non-novel research, the increasing use of bibliometric indicators for evaluation may bias funding decisions against novelty.

In Wang, Veugelers, and Stephan (2016), we examine the complex relationship between novelty and impact. To do this, we develop a measure of novelty for individual scientific publications , using the life-time citation trajectories of 1,056,936 research articles published in 2001 across all disciplines in the Web of Science (WoS).

To assess the novelty of a research article, we examine the extent to which it makes novel combinations of prior knowledge components. Specifically, we examine whether an article makes new combinations of referenced journals that have never appeared in prior literature, taking into account the knowledge distance between the newly-combined journals based on their co-cited journal profiles (i.e., “common friends”). Technically, we measure the novelty of a paper as the number of new referenced journal pairs weighted by the cosine similarity between the newly-paired journals.

Fig 1. Impact profile of novel research.

impact novel research

(A) Estimated dispersion of citations (14-year).  (B) Estimated probability of being among the top 1% cited articles in the same WoS subject category and publication year, based on 14-year citations.  (C) Estimated probability of being cited by big hits.  (D) Estimated probability of being among the top 1% cited articles in the same WoS subject category and publication year, based on 14-year home and foreign field citations separately.  (E) Estimated probability of big hits, using 14 consecutive time windows to dynamically identify big hits.  As an example, big hits in year 3 are identified as the top 1% highly cited papers based on their cumulative citations in a 3-year time window, i.e., from 2001 to 2003.  (F) Estimated Journal Impact Factor.  All regression models control for field (i.e., WoS subject category) fixed effects, the number of references and authors, and whether internationally coauthored, (C) additionally controls for the number of citations.  All estimated values are for an average paper (i.e., in the biggest WoS subject category, not internationally coauthored, and with all other covariates at their means) in different novel classes.  The vertical bars represent the 95% confidence interval.  Data consist of 1,056,936 WoS articles published in 2001 and are sourced from Thomson Reuters Web of Science Core Collection.

Applying this newly-minted indicator of novelty, we find that novel research is relatively rare:  Only 11% of all papers make new combinations of referenced journals, and the degree of novelty is also highly skewed. To work with this skewedness, we classify papers into three categories: (i) non-novel, if a paper has no new journal combinations, (ii) moderately novel, if a paper makes new combinations but has a novelty score lower than the top 1% of its field, and (iii) highly novel, if a paper has a novelty score among the top 1% of its field.

In view of the risky nature of novel research, we expect novel papers to have a higher variance in their citation impact. We estimate the the mean and dispersion of received citations in a 14-year time window for novel papers compared to non-novel ones, controlling for other potentially confounding factors, such as the scientific field of the articles. We find that the dispersion of the citation distribution is 15% higher for highly novel papers than non-novel papers, confirming their higher risk profile (Fig 1A).

We also find that novel research papers are more likely to become “big hits,” i.e., being top 1% highly cited among the articles in the same field and publication year:  The odds of being a big hit are 40% higher for highly novel papers than comparable non-novel papers (Fig 1B). Novel research is not only more likely to become a big hit itself but also more likely to stimulate follow-on research which generates major impact: The odds of being cited by other big hits are 26% higher for highly novel papers than for comparable non-novel papers receiving the same number of citations (Fig 1C).

Examining the disciplinary profile of impact reveals that, compared with an equally cited non-novel paper, highly novel papers are cited by more fields and have a larger ratio of citations from other-than-own fields, suggesting a greater cross-disciplinary impact of novel research. Furthermore, we find that novel papers are more likely to become highly cited only in foreign fields but not necessarily in their home field (Fig 1D), calling to mind the passage from Luke 4:24:  “Verily I say unto you, No prophet is accepted in his own country.”

We find that novel papers have a delayed recognition. Highly novel papers are less likely to be top cited in the first 3 years after publication but significantly more likely from the fourth year onward, and their advantage over non-novel papers further increases over time (Fig 1E).  Moderately novel papers suffer even more from delayed recognition: They only have a significantly higher chance of being top cited with time windows of at least 9 years. The finding of delayed recognition for novel research suggests that standard bibliometric indicators which use short citation time-windows (typically two or three years) are biased against novelty.

Assessing the Impact Factor of the journal in which the paper is published, we find that of highly-novel papers to be 18% lower than comparable non-novel papers (Fig 1F), suggesting that novel papers encounter obstacles in being accepted by journals holding central positions in science.

Taken together, our results suggest that standard bibliometric measures are biased against novel research. Consequently, they may fail to identify papers and individuals doing novel research. This bias against novelty matters, because novel research, as we have shown, is much more likely to become a big hit in the long run, as well as to stimulate follow-up big hits.

The bias against novelty is of particular concern given the increased reliance funding agencies place on classic bibliometric indicators in making funding and evaluation decisions. This bias against novelty applies not only to funding decisions but to science policy more generally. The prevailing use in various science decisions of standard bibliometric indicators which rely on short citation time windows and Journal Impact Factors is likely to disincentivize novel research.

In addition, the finding that novel papers are highly cited in foreign fields highlights the importance of avoiding a monodisciplinary approach in peer review (see Gabriele Bammer’s recent piece for more on this). Peer review is widely used in science decision-making and is typically organized along disciplinary lines. Studies of interdisciplinary research demonstrate that a discipline-based science evaluation system is detrimental to the advancement and societal accountability of science. Our findings suggest that peer review which is bounded by disciplinary boarders may also fail to recognize the full value of novel research which typically lies outside its own field.

If we want to encourage risk taking, we need to be aware of the inherent bias in bibliometric measures against novel research. More importantly, we advocate, when bibliometric indicators are used as part of the evaluation process, a wider portfolio of indicators and of time windows beyond the standard window of two or three years.

This blogpost is based on Wang J, Veugelers R, Stephan P (2016) Bias against novelty in science:  A cautionary tale for users of bibliometric indicators. CEPR Discussion Paper No. DP11228; NBER Working Paper No. 22180.

Featured image credit: Lego figures depicting breaking through a wall by Brian Rinker licensed under the terms of the cc-by-2.0 via Flickr.

Note: This article gives the views of the author, and not the position of the LSE Impact blog, nor of the London School of Economics. Please review our Comments Policy if you have any concerns on posting a comment below.

About the Authors

Jian Wang is a postdoctoral fellow at the Department of Management, Strategy and Innovation and the Center for R&D Monitoring, KU Leuven (Belgium).  His postdoctoral research is funded by the Research Foundation – Flanders (FWO).  He is specialized in science and technology policy.

Dr. Reinhilde Veugelers is a full professor at KULeuven (BE) at the Department of Management, Strategy and Innovation.  She is a Senior Fellow at Bruegel since 2009. She is also a CEPR Research Fellow, a member of the Royal Flemish Academy of Belgium for Sciences and of the Academia Europeana. From 2004-2008, she was on academic leave, as advisor at the European Commission (BEPA Bureau of European Policy Analysis). She was the President-Elect of EARIE (European Association for Research in Industrial Economics).  She currently serves on the ERC Scientific Council.  She is a member of the RISE Expert Group advising the EU Commissioner for Research and Innovation.

Paula Stephan is a professor of economics, Georgia State University and a Research Associate, National Bureau of Economic Research.  She is a Fellow of the American Association for the Advancement of Science.  Her research focuses on the careers of scientists and engineers.  Her 2012 book How Economics Shapes Science was published by Harvard University Press.

Print Friendly, PDF & Email

About the author

Blog Admin

Posted In: Academic publishing | Citations | Evidence-based research

1 Comments