A new metric offers insight into the societal impact of scholarly research by tracking the mentions of academic publications in policy documents. Lutz Bornmann, Robin Haunschild and Werner Marx have studied the usefulness of this metric, taking climate change research as their example, and found only a low percentage of papers were referenced in the relevant literature. Does this mean the research is not relevant? Or is it rather a reflection of poor levels of interaction between academics and policymakers? And what else does the data tell us about the likelihood of academic research being cited in policy literature?
In recent years, societal impact measurements of academic research have become more and more important. This trend is not only visible by their consideration in national evaluation systems (e.g. the UK’s Research Excellence Framework), but also in the commercial success of providers delivering altmetrics data (e.g. Altmetric) which propose that altmetric scores can be used to measure societal impact.
In our recent study we dealt with a relatively new form of societal impact measurement. Recently, Altmetric developed a text-mining solution (the Altmetric Policy Miner) to discover mentions of academic publications in policy documents, in order to uncover the interaction between science and politics. On one side, governments allocate very large amounts of public money to various units (e.g. researchers or institutions) for various forms of research. In many countries, the public money is distributed by the soft money system whereby researchers formulate proposals for projects and funding bodies decide whether to accept or reject them. For governments, academic science is one section of a vaguely defined research and development (R&D) system which ranges from basic science to near-market technological development. On the other side, independent and still-active scientists advise stakeholders in the policy area. Scientific advice can be made in direct face-to-face interactions or indirectly, as papers written by scientists (actually written for their peers) are read by political actors and mentioned in policy-related documents. The latter type of interaction can possibly be measured by the new data source offered by Altmetric.
Image credit: Global Warming by Paul VanDerWerf. This work is licensed under a CC BY 2.0 license.
Taking a comprehensive dataset of publications on climate change as our example, we studied the usefulness of Altmetric’s new data source for measuring societal impact. We chose climate change literature because:
- corresponding policy sites are continuously analyzed by Altmetric
- we expected to observe many references to the scientific literature in policy documents as climate change has been a policy-relevant topic for many years
- policy documents may be a good proxy for impact on the section of society related to politics.
We were especially interested in the characteristics of the papers mentioned in the policy documents: are these papers published in certain journals (e.g. popular journals like Nature and Science), in certain publication years (e.g. more recent years), or with certain document types (e.g. reviews)?
We constructed a set of 222,060 papers (articles and reviews only) on climate change published between 1980 and 2014 via a sophisticated search method called ‘interactive query formulation’ (a detailed explanation of the procedure is available). The DOIs (n=191,276; 86.1% of the full publication set) were used to retrieve information about policy mentions from Altmetric via their application programming interface (API). For only 1.2% (n=2,341) of the papers were we able to find at least one policy mention. This low percentage might be due to the fact that Altmetric only recently started to analyze policy documents and the coverage of the literature remains low (but is to be extended). However, the low percentage might also demonstrate that only a small part of the literature is really policy-relevant and most papers are relevant only to other researchers studying climate change. Other reasons for the low percentage might be that policy documents do not mention every important paper on which a policy document is based, or that there are possible barriers and low interaction levels between researchers and policymakers.
The low number of mentions in policy documents further raises the question of what this particular metric really measures. Is it the relevance of academic papers? Do the mentions reflect the efforts of researchers to interact with policymakers? An ongoing relationship between researchers and policymakers? Or the efforts of policy organizations to reference research in policy documents?
In order to find out which types of papers are more or less interesting in the policy context we compared the distribution of papers among all climate change papers (CCP) with those climate change papers mentioned at least once in policy documents (CCP_P). The results showed that the policy literature tends to cite research published a longer time ago than researchers do in their papers. Thus, research papers seem to need more time to produce impact on policy than on research itself.
As expected, review articles are overrepresented among CCP_P: their observed CCP_P value is higher than the expected value delivered by the CCP distribution. Review articles summarize the results of many primary research papers and connect research lines from different research groups. In our study, we further revealed that papers published in high-impact journals such as Nature and Science as well from the areas ‘Earth and related environmental sciences’ and ‘Social and economic geography’ are especially relevant in the policy context.
Our study was a first attempt to study mentions of scientific publications in policy documents; we would encourage further empirical studies. It will be interesting to see whether more papers are used in policy documents in upcoming years (as Altmetric’s coverage of the policy literature increases). Furthermore, it would be interesting to generate results from other policy-relevant fields of research in order to draw comparisons with climate change research. In other fields of research, will more or less than 1.2% of publications be mentioned in policy documents? Do policy documents from other fields focus more on recent literature than is the case with climate change? We await the findings with interest.
This blog post is based on the authors’ article, ‘Policy documents as sources for measuring societal impact: how often is climate change research mentioned in policy-related documents?’, published in Scientometrics (DOI: 10.1007/s11192-016-2115-y).
Note: This article gives the views of the author, and not the position of the LSE Impact Blog, nor of the London School of Economics. Please review our comments policy if you have any concerns on posting a comment below.
About the authors
Lutz Bornmann works as a sociologist of science at the Division for Science and Innovation Studies in the Administrative Headquarters of the Max Planck Society in Munich, Germany.
Robin Haunschild is a chemist and works as a bibliometrician in the Scientific Facility for Information Retrieval of the Chemical Physical Technical section of the Max Planck Society at the Max Planck Institute for Solid State Research in Stuttgart, Germany.
Werner Marx is retired, but is still working as an information specialist and guest at the Max Planck Institute for Solid State Research in Stuttgart, Germany.