The process by which research gets put into action is far from clear cut, argues Stacy Konkiel. Extracting references to research from policy documents is a step towards illuminating the murky path. But we should be careful not to disregard other forms of evidence like online and media mentions as they are closely interrelated and may even lead to quicker impacts upon public policy.
Forget the Ivory Tower stereotype you might know: the majority of researchers want to have an impact upon public policy. That’s according to one study of public health researchers: 86% of those interviewed agreed that their most important task “is to produce research that can further public health policy and/or practice” (Haynes 2011).
This attitude can be seen increasingly in online discussions, too, where many researchers–particularly in the social sciences–are reflecting critically upon what it means to be an engaged scholar. This attitude may, in part, be due to the rise of the “impact agenda” at many universities worldwide, whereby researchers are increasingly expected to produce research that has “real world” implications. Results-driven initiatives like the 2014 REF and similar exercises have made it clear: no longer is being a prolific, highly cited author enough; you must have an influence on the lives of the public, as well.
So, how can a researcher learn about their impacts upon public policy, especially in cases where they may not have had a direct relationship with policymakers? I’d argue that tracking the impact of research upon policy can and should be done through analyzing citations in public policy documents. And, backed by the promising results of an exploratory study I recently presented at the 2015 ASIS&T SIG/MET workshop, I believe that researchers who wish to have an impact upon public policy can use social media tools like Twitter to raise their profile and expedite their policy impacts.
Before we begin, let’s take a step back: what does it mean for research to have an impact upon public policy?
How research influences public policy
In the past, it was difficult to know how and when research influenced policy. Weiss (1979) once argued that “most studies appear to come and go without leaving any discernible mark on the direction or substance of policy”. But the times, as they say, are changing. According to one recent study, approximately one in four policymakers use academic research to inform their work on a daily or weekly basis (Zardo & Collie, 2014), and the ways in which they put that research into action are varied (more on this below).
That said, the process by which research gets put into action is far from clear cut. As Haynes et al (2011) point out, “Linear, rational depictions of the ‘the policy cycle’ are increasingly viewed as idealised normative models that poorly describe a far messier process.” Indeed, Weiss (1979) describes several processes by which research is enacted into policy, the most common of which is practically osmosis. In her article, she describes the Enlightenment Model of knowledge utilization, by which social science research influences policymakers in a way that the policymakers themselves may not even realize. Research might introduce general ideas or ways of thinking about problems that are influential in such an indirect way that researchers may not be credited–often because policymakers don’t even know they’ve been influenced at all!
Tracking research impacts on the social web
The murky path from research to policy, summarized in Landry, Lamari & Amara’s 2003 adaptation of Knott and Wildawsky’s “Stages of Knowledge Utilization” (seen below), has a lot to do with the difficulty of tracking awareness of research. Much of the battle of getting research put into policy (Stages 4-6) lies simply in making policymakers aware that such research even exists (Stage 1). Such awareness often comes via the mainstream media (Tieberghien, 2014), making the media a powerful platform for researcher outreach strategies. Other effective ways of getting research into practice include fostering relationships with policymakers and the creation of translational outputs (lay readings, protocols, training materials, etc) based upon research findings (Newson et al, 2015; Cohen et al 2015).
Thanks to the social web, it’s now possible to track research’s journey along each of these stages in a way that couldn’t have been imagined by Knott and Wildawsky back in 1980. We can use text mining to:
- discover when research is discussed in news articles;
- see the traces of researchers’ engagement with policymakers in their conversations posted publicly on listservs and Twitter; and
- track downloads and pageviews for non-traditional research outputs like protocols, white papers, and the like, to understand how widely read translational outputs are.
This type of impact data is often referred to as “altmetrics”.
Most importantly, we can extract references to research from public policy documents; this is perhaps the most concrete way of understanding impact. There are three major themes for citations to research in public policy documents, according to Weiss in Haynes et al (2011):
- data used instrumentally to inform policy decisions;
- argument used strategically to address values and interests or symbolically to support, justify or refute predetermined positions; or
- ideas that influence the policy climate and agenda-setting by illuminating, supporting and challenging existing paradigms.
These three types of citations are the ultimate pieces of evidence that researchers should use to discover the influence they’ve had upon “the real world” via public policy. But we should be careful not to disregard other impact evidence like the altmetrics described above, because they’re closely interrelated with and can even lead to quicker impacts upon public policy.
“High attention” research gets cited more quickly and often in policy
Initial research has shown that research outputs cited in policy that receive a high amount of attention online (as measured by being in the top 0.25% of outputs ranked by Altmetric score) tend to get cited more quickly. The mean time to first policy citation for “high attention” research is 1.7 years, compared with 3.4 years for other research discussed online.
“High attention” research also receives slightly more policy citations (1.18 citations per article, compared to 1.09 citations per article). To be clear, this is an exploratory analysis with small sample sizes (n=116 for both), but the preliminary data looks promising.
What do these findings mean for the average researcher interested in making an impact upon public policy? Simply put: making a splash online and in mainstream media can get the attention of policymakers and potentially put your important discoveries into practice faster. The latter point–speed–is especially important for time-sensitive policy issues (e.g. responding to the international refugee crisis, curbing the spread of the Zika virus, and so on). Tips on how to promote your work to the right audiences can be found in the Impactstory e-book, The 30-Day Impact Challenge.
Academia’s still got a long way to go to if we’d like for all relevant research to get the policy impacts it deserves. Many researchers aren’t recognized by the powers that be for the impacts that their work has had in non-scholarly contexts. The incentives to do outreach simply don’t exist. Luckily, many scholars aren’t waiting around for the system to change. To explore more about your research’s impact upon public policy–alongside other evidence of discussions and reuse on the social web–check out the free Altmetric bookmarklet.
Disclosure: The author is an employee of Altmetric.com, an altmetrics service provider. Altmetric is a Digital Science company.
Note: This article gives the views of the author(s), and not the position of the LSE Impact blog, nor of the London School of Economics. Please review our Comments Policy if you have any concerns on posting a comment below.
Stacy Konkiel is Research Metrics Consultant with @altmetric.