Outside of specific institutional and organizational settings discussions about ‘impact’ often descend into confusion, as the term holds several meanings for researchers. Responding to a provocation from Ziyad Marar “On Measuring Social Science Impact” and reflecting on different ways of understanding impact, Anne-Wil Harzing, provides a guide to understanding different aspects of impact for academics and a set tools for how they can assess impact in different ways.

It should be easy, shouldn’t it? Impact is impact is impact. Well, no… When talking about research impact, academics often have very different understandings of the concept. Worse still, they might not even be aware that different interpretations are possible.

As a result, any discussion about research impact soon descends into a Babylonian speech confusion. It also means that many academics are struggling to show the impact of their research, for instance when making a case for promotion (as I discuss in a recent blogpost). Here, I want to provide a brief clarification of the concept of research impact and show how it differs by academic role.

What is impact?

The Oxford dictionary defines impact as “a marked effect or influence.” Research impact thus means that research has affected or influenced something or someone. Unfortunately, this immediately throws up even more questions:

• Whom has it impacted, i.e., who is the target audience?
• How has it made an impact, i.e., what was its ultimate goal?
• Through what means has this impact occurred, i.e., what are the primary outlets?
• How do we know this impact has occurred, i.e., how can we measure it?

The answer to these four questions very much depends on the specific academic role we are looking at (See table).

How research impact differs by academic role:

Research role

In the research role of academic jobs, the target audience is other academics, and the ultimate goal is to progress scientific knowledge by incorporating research into a wider body of scholarly knowledge.

Researchers connect with other academics through academic journals, research monographs, and conferences. The preferred types of outlets are highly dependent on disciplinary norms and preferences. In the Life Sciences and Natural Sciences academic journals “rule.” In the Humanities and some of the Social Sciences, books remain a popular medium (a research strategy discussed here). In some engineering disciplines, conference papers are a popular way of diffusing knowledge quickly.

Whether or not we have influenced other academics is typically measured by citations in academic journals, books, and conference papers. Although we know that academics are sometimes careless in their referencing and there are many reasons to cite papers, one would normally expect citations to signify at least some level of impact on the citing academic. Citations can also be field normalized to account for differences in citation practices across disciplines.

Teaching role

Academics also have a tremendous impact on students. Obviously, some of this impact is unrelated to research. However, in any good university, research feeds into the classroom and students benefit from research-informed teaching, allowing them to develop critical thinking skills.

As academics, we facilitate this directly through prescribing our own and other academics’ research as course readings. But in many cases, we will need to “translate” this research to make it more accessible for a student audience. We do so through publishing textbooks or articles in practitioner journals such as (if you are fortunate) Harvard Business Review.

Image Credit: Inside Bruegel / Kunsthistorisches Museum Wien / The Tower of Babel / Pieter Bruegel, by Ars Electronica / Robert Bauernhansl, via Flickr (CC BY-NC-ND 2.0).

So how do you know whether your research is used beyond your own classroom? You can find out if your research is cited in textbooks by using Google Books. To discover whether your publications (both academic articles and textbooks or practitioner articles) are listed in teaching syllabi, Open Syllabus Explorer (See Open Syllabus Explorer: evidencing research-based teaching?) is an incredibly useful tool.

Engagement role

So far, we have discussed the two key functions of any university: research and teaching. But there is also a third function: external engagement. This captures the impact of research on industry, government, and the public/society at large, with the ultimate goal being to address key societal problems. It is the kind of impact incorporated in Impact Case Studies in the UK Research Excellence Framework (REF), defined as “an effect on, change or benefit to the economy, society, culture, public policy or services, health, the environment, or quality of life, beyond academia”.

Making research accessible to an audience outside academia typically means “translating” research for non-academic use by writing articles for practitioner / professional journals and magazines, as well as publishing policy reports. It may involve the use of (social) media, not just to diffuse already published work, but to allow continued engagement of non-academic audiences in research. Integration of non-academic stakeholders from the inception of research projects is increasingly common in the social sciences.

Measuring this type of impact is challenging. Recent efforts like Overton – a database that captures policy impact by tracking citations in policy documents – might help. Academics may also be able to evidence changes in government policy or legislation that are linked to their research. A more general case for impact can be made by referring to frameworks such as the United Nations Human Development Index and Sustainable Development Goals.

Horses for courses

Different universities have very different strategies for research impact, especially in the research and engagement roles. In the 2021 REF, the national research evaluation in the UK, the field of Business & Management showed a very different ranking of universities depending on which of the constituent criteria you focused on. London Business School, London School of Economics, and University College London excelled in the perceived quality of their publications, but only had an average performance when looking at their societal impact. In contrast, Middlesex University, the University of Westminster – both modern universities – and SOAS (the School of Oriental and African Studies) topped the list for societal impact. However, they only had an average performance when looking at the perceived quality of their publications.

In sum: measuring impact

It is almost impossible to measure the ultimate effect for these three areas of research impact – progressing scientific knowledge, developing critical thinking, and addressing societal problems – accurately and comprehensively by using citations or other quantitative metrics alone. On the other hand, relying only on testimonials and a fully narrative approach to establishing research impact is unlikely to convince audiences, either. So in evidencing impact – for instance when making a case for promotion – academics are advised to rely on a combination of metrics, qualitative evidence such as testimonials, and career narratives.

Readers may also be interested in these further resources highlighted by the author: How to Improve your research profile, impact, and reputation, How to write Effective promotion applications, How to use Social media to support your career. You can find all of Anne-Wil Harzing’s LSE Impact Blog posts here. A version of this post first appeared on Social Science Space as: Impact ≠ Impact ≠ Impact.

The content generated on this blog is for information purposes only. This Article gives the views and opinions of the authors and does not reflect the views and opinions of the Impact of Social Science blog (the blog), nor of the London School of Economics and Political Science. Please review our comments policy if you have any concerns on posting a comment below.

Image Credit: Inside Bruegel / Kunsthistorisches Museum Wien / The Tower of Babel / Pieter Bruegel, by Ars Electronica / Robert Bauernhansl, via Flickr (CC BY-NC-ND 2.0).