Searching for anything in the vast chaos of cyberspace can be a major case of needles and haystacks. Alistair Scott describes how his team has established systems for capturing data related to research outputs. By using carefully constructed search strings (combinations of search terms) and a bit of fine-tuning, you can map and visualise online conversations and the progression of real-world impacts.
How is your research being used? Who’s talking about it? Today researchers and their institutions, as well as donors, the media and even the general public are increasingly demanding answers to these questions. But as we all know, discovering what happens to your research after publication can be something of a challenge. It can be difficult even to get a sense of whether people are thinking about or discussing your research, let alone whether your work is having any big, real-world impact.
Thanks to the Internet however, and especially to that sub-set of the internet known as social media, there’s a good chance that if people are talking about your research at all, some of those conversations are likely to be taking place online and in the public eye. And when this happens it usually leaves some kind of digital trace which can in principle be collected and analysed. Indeed because the very nature of social media it is theoretically possible not just to find isolated citations, but also conversation threads, where your ideas are being discussed and developed within a wider context, just as they might in a seminar or conference in the real world. They are effectively narratives; real-world stories about other people’s reactions to your research.
Image credit: Clickpath Analysis by KTB (Wikimedia, CC BY-SA)
So how might we find and track these online conversations? Tuning in to the digital debate
Many commercial applications exist which might, in principle, help with this. Media monitoring services, citations-tracking tools and alternative metrics applications are all available. And they all allow you to search for instances where research publications have been cited or referred to in online media. Powerful though these applications often are, however, they’re not generally optimised for tracking how research ideas are actually being discussed. Persuading them to do this can require some additional effort.
At IDS our approach has been to experiment with different combinations of software applications, including some high-end media monitoring systems such as Meltwater and Vocus, but also some of the many powerful low-price applications, such as Zapier. The starting point with all these applications is to carry out an initial search for mentions of the research you want to track. As we all know of course, searching for anything in the vast chaos of cyberspace can be a major case of needles and haystacks. Identifying search terms that actually zone in on what you’re looking for often requires considerable trial and error, and it is often more of an art than a science, not least because every search is different and there are no hard and fast rules about what works and what doesn’t. But by using carefully constructed search strings (combinations of search terms) and fine-tuning them when they don’t work (which is usually the case at first) you can often find what at least some of what you’re looking for. This data can then be easily exported, using basic spreadsheet applications such as Excel or Google docs, and the results can be displayed graphically, using any number of visualisation applications, such as Tableau or even Excel itself.
Mapping online conversations
The result of all this is a timeline, showing a chronological sequence of articles, tweets, Facebook posts, blogs or e-discussion posts, each of which refers, in some way, to the research you’re trying to track. And if you’re lucky, some of this chronology will show the development of ideas over time, as different people, in effect, debate the research over time, from different online platforms.
The timeline below provides an example. It displays both relevant pieces of text which comment or draw on the research (in this case, comments on IDS’s Hunger and Nutrition Commitment Index), together with profile information provided by each contributor:
A computer generated map of Twitter exchanges about the IDS Hunger and Nutrition Commitment Index using Zapier for tracking and Tableau for visualization.
In the case of Twitter, for example, the timeline displays the author’s Twitter profile which, depending on the individual contributor’s preferences, often contains a brief description of their professional interests, job title, host organisation, geographical location and how many Twitter followers they have.
In this way the timeline not only shows what has been said, but also some details about the people saying it. It can therefore provide a sense of whether a given contributor is broadly speaking a researcher, a practitioner, an activist, a policy adviser or some other role.
This type of visualization is of course, unlikely to provide a definitive illustration of the real-world impact of your research. But it may at least reveal something useful and interesting about the reach of that research and the way that people are engaging with it. If your research relates to hunger and nutrition in East Africa for example, it might be good to know that it’s been picked up and discussed by CSOs and NGOs involved in lobbying to improve nutrition in Tanzania. Or that it’s been discussed in an e-discussion by health and nutrition consultants whose profiles suggest they work closely with relevant governments. If nothing else it can demonstrate that your research is being talked about in the world beyond academia.
For us at IDS, these visualisations are very much a first step. Our aim is to develop an actual open source web application which can carry out this kind of analysis in a practical and ergonomic format. We are interested in exploring the potential for collaborating with others working along similar lines elsewhere in the academic and research worlds. So regardless of your particular academic sector, if this is of interest then do please get in touch.
Note: This article gives the views of the author, and not the position of the Impact of Social Science blog, nor of the London School of Economics. Please review our Comments Policy if you have any concerns on posting a comment below.
Alistair Scott works in the Knowledge Mobilisation & Impact team at the Institute of Development Studies in Brighton, UK. He works on various knowledge related projects and is currently exploring methodologies for tracking and monitoring research uptake in social media and on the Internet. With a PhD in Human-Computer Interface Design his background is in designing websites and multimedia applications for online research communication and e-Learning, as well as Open Access advocacy and NGO campaigning. Prior to joining IDS he worked variously as an ergonomist, research fellow in the Open University’s Institute of Educational Technology, Oxfam Campaigns Officer, and Technology Officer and Trainer for OneWorld.Net. And as an amateur song-writer in a Brighton rock group he recently made an inaugural pop video.