The suitability of altmetrics for use in assessments of societal impact has been questioned by certain recent studies. Ismael Ràfols, Nicolas Robinson-García and Thed N. van Leeuwen propose that, rather than mimicking citation-based approaches to scientific impact evaluation, assessments of societal impact should be aimed at learning rather than auditing, and focused on understanding the engagement approaches that lead to impact. When using altmetric data for societal impact assessment, greater value might be derived from adopting “interaction approaches” to analyse engagement networks among researchers and stakeholders. Experimental analyses using data from Twitter are presented here to illustrate such an approach.

In response to increasing policy demands for methodologies to assess societal impact, the use of altmetrics has been put forward as a possible approach. However, this approach has been questioned by studies analysing what altmetric indicators actually capture. Experts such as Cassidy R. Sugimoto have argued that “media has opened a new channel for informal discussions among researchers, rather than a bridge between the research community and society at large”, adding that social media attention should not be confused with impact. A review by Haustein et al concluded that altmetric indicators can signal a variety of phenomena – including early scientific impact, educational and practical use, popularity, buzz, visibility, or societal impact – depending on the type of social media and the social contexts analysed. Given these ambiguities, can altmetric data be useful for assessing societal impact?

We propose that it could – but only if we radically shift the purpose of evaluations. Currently, many evaluations of scientific impact or “excellence” follow a logic of audit or accountability which is consistent with bibliometric indicators based on citation counts. Most current altmetric indicators are novel because they use social media, but tend to follow a traditional bibliometric approach based on citation counting, and implicitly appear to assume that the purpose of assessment is accountability under a productivity framing of evaluation; i.e. the more (papers, citations, tweets, blogs, etc.), the better. However, the large variety of social media channels and their unevenness in coverage means that altmetric data cannot be used in the same, supposedly generalisable, manner commonly applied in bibliometrics (which is, in fact, also disputed and perhaps having undesired effects).

In contrast, carefully designed approaches to societal impact assessment, such as ASIRPA, SIAMPI or PIPA, adopt the logic of learning rather than audit. Notice our focus here is on the “societal” part of impact only – broadly understood as the generation of innovations or behavioural changes. Since research is only one factor in a network of determinants of societal impact, it seems pointless to focus on assessing how much impact can be attributed to science. Flour and eggs contribute towards making a tasty cake, but does it make sense to attribute a measure of such contribution to each separate ingredient? The cake results from the mixing and the heating; its tasty value derives as much from the cooking processes as from the ingredients. And more is not always better; a pinch too much of salt or lemon or cinnamon, or a slight increase in temperature, will spoil the taste!

This is why approaches like SIAMPI and ASIRPA emphasise the importance of understanding the interactions among researchers and societal actors: because the interaction processes constitute the mechanisms leading to societal impact. For the sake of clarity, we call these methodologies “interaction approaches”. The goal of such assessment approaches is to learn how the social engagements of researchers can support and improve societal impact. To do so, interaction approaches carry out case studies in which they investigate the networks that result in impact. This focus on context, engagement and networks stands in contrast to citation approaches’ focus on outputs and counting, as illustrated in Table 1.

Table 1: Differences between citation and interaction approaches to societal impact assessment.

In a recent article, we propose that for social media data to help in mapping societal impact, altmetrics should follow interaction approaches instead of emulating the indicators of scientific impact based on citations. This means that altmetric analyses could be helpful for assessing societal impact when embedded within larger methodologies aimed at learning and focused on engagement. While not sufficiently robust to produce reliable insights on its own, altmetric data can be used as complementary and supporting evidence within case studies based on diverse data sources. Thus it can help in triangulation of evidence or as an illustration of insights obtained from more reliable methods (e.g. ethnographies).

We present an example of this “alternative” (!) approach to altmetrics by mapping the networks of interaction among researchers and stakeholders on social media platforms. This is a shift from current altmetric analysis, based on mentions of publications (counting outputs), to analysing relations among actors (networks of processes). Using Twitter data, we analyse the interaction community of a researcher by selecting nodes that are both her followers and followees. Second, we build the network of follower-followee links within her interaction community, as shown in Figure 1.

Figure 1: Interaction networks of two researchers (names not shown) derived using Twitter data of their reciprocal follower-followee relations. The nodes are characterised by the type of institutional affiliation. Source: Robinson-Garcia, Nicolas and van Leeuwen, Thed N. and Rafols, Ismael, ‘Using Almetrics for Contextualised Mapping of Societal Impact: From Hits to Networks‘ (March 14, 2017). Copyright © The Authors; reproduced here with permission.

Through these networks we aim to understand researchers’ contexts and their potential engagement. Researcher 1 exhibits a cohesive community which is mainly international and exclusively academic. The network of Researcher 2 contains local stakeholders from different sectors (45% non-academic), with presence of public and private sector. From complementary data sources, we learned that Researcher 1 has a mainly academic focus, though she does conduct some private consultancies (attention: these consultancies were not captured by Twitter!); whereas Researcher 2 is strongly embedded in a science policy community, as suggested by the network analysis.

The examples illustrate that, while the networks do not directly reflect societal engagement, they can suggest the contexts of interactions between researchers and stakeholders. This information about contexts can be useful to guide further enquiries, helping select the type of quantitative or qualitative approaches that are appropriate to gain a deeper understanding of the case studies. The network visualisation facilitates a conceptual shift from a linear assessment of attribution towards an assessment of contribution to impact.

In summary, interaction approaches such as SIAMPI or ASIRPA have highlighted that societal impact assessments should not be carried out within the audit framework often applied to them. The lack of robustness, the variability and the uneven coverage of altmetrics make them particularly problematic for assessing societal impact under citation approaches. However, the diversity of altmetric data also makes it valuable for capturing traces of engagement that no other information source can provide. Interaction approaches to altmetrics, developed with networks analysis or other novel visualisation methods (e.g. see Costas et al), can help altmetrics escape policy pressures pushing for audit assessments and enable the use of social media data in a meaningful and responsible way. We hope this can be an example of how a “new generation of metrics” can serve the priorities of open science, as proposed in this week’s Report of the EC Expert Group on Altmetrics.

This blog post is based on the authors’ article, “Using Almetrics for Contextualised Mapping of Societal Impact: From Hits to Networks’” currently available as a preprint on SSRN.

Featured image credit: bol crown kite 7 by Katrina Br*?#*!@nd (licensed under a CC BY-SA 2.0 license).

Note: This article gives the views of the authors, and not the position of the LSE Impact Blog, nor of the London School of Economics. Please review our comments policy if you have any concerns on posting a comment below.

About the authors

Ismael Ràfols is a research fellow at Ingenio (CSIC-UPV), a science policy institute at the Universitat Politècnica de València. He is also affiliated to the SPRU (Science Policy Research Unit), University of Sussex. Ismael’s research focuses on plural and inclusive use of S&T indicators for informing evaluation, funding and research strategies. His ORCID iD is 0000-0002-6527-7778.

Nicolás Robinson-Garcia is a research fellow at Ingenio (CSIC-UPV), a science policy institute at the Universitat Politècnica de València. His research interests are the understanding of the different roles of researchers and the analysis of research evaluation schemes based on quantitative indicators. His ORCID iD is 0000-0002-0585-7359.

Thed van Leeuwen is a senior researcher at the Centre for Science and Technology Studies (CWTS) of Leiden University. The main focus of his research centres around the organisation of research, the various communication practices in the social sciences and the humanities, and the way this is evaluated.

 

Print Friendly