LSE - Small Logo
LSE - Small Logo

Bella Reichard

Mark Reed

Jennifer Chubb

Ged Hall

Lucy Jowett

Alisha Peart

Andrea Whittle

May 11th, 2020

The Grammar of Impact – What can we learn from REF 2014 about writing impact case studies? 

0 comments | 69 shares

Estimated reading time: 7 minutes

Bella Reichard

Mark Reed

Jennifer Chubb

Ged Hall

Lucy Jowett

Alisha Peart

Andrea Whittle

May 11th, 2020

The Grammar of Impact – What can we learn from REF 2014 about writing impact case studies? 

0 comments | 69 shares

Estimated reading time: 7 minutes

Impact case studies will form an important part of all universities’ eventual submissions to the next round of the Research Excellence Framework (REF). Drawing on a linguistic and thematic analysis of 175 impact case studies from REF 2014, Bella Reichard, Mark Reed, Jenn Chubb, Ged Hall, Lucy Jowett, Alisha Peart and Andrea Whittle set out the written characteristics that defined high-quality impact case studies in REF 2014 and suggest how they can inform the creation of better impact case studies as part of the upcoming REF. 


Writing impact case studies for REF2021 is something that many people in the UK HE sector are currently involved in. One way to get inspiration is to check case studies from REF2014, in particular those that were clearly given 4*. We were interested to what extent you can generalise from these examples, and how they differ from low-scoring case studies. In a recent article, we conducted a comprehensive analysis of 175 REF2014 case studies across each of the Main Panels. To do this, we used qualitative and quantitative linguistic analysis of high versus low-scoring cases to provide evidence of what it took for a case study to get a top score in 2014. Although scores were not made public for individual cases, we looked for institutions whose case studies were all given grades in the same range to identify 175 (out of 7,000) high- or low-scoring case studies from a cross-section of disciplines. 

In a nutshell, high-scoring case studies clearly articulated evidence of significant and far-reaching benefits that could be clearly attributed to research conducted at submitting institutions. 

However, new rules are being introduced for the upcoming REF and there are fears that the bar may have been raised. This could mean that sticking to what we understand was ‘best practice’ during REF 2014 may not be enough to reach top scores in 2021, so the lessons from this research should be seen as the minimum required to write a top-scoring case study for REF2021. Also keep in mind that while there are examples of 2* and 3* impact case studies that might have scored higher if they had been better written and evidenced, it is not possible to get top grades unless you actually have significant and far-reaching impact. Here are some key lessons from what we found in our research to help boost the articulation of impact in REF2021.

 

 

Key lessons from high-scoring case studies in REF2014

 

1. Establish links between research (cause) and impact (effect) convincingly 

In REF2014:

  • Only 50% of low-scoring case studies clearly linked the underpinning research to claimed impacts (compared to 97% of high-scoring cases). 
  • High-scoring case studies were significantly more likely to include phrases that attributed impact to research: “cited in (policy documents)”, “used to (inform)” and “resulted in”. Those phrases indicate a focus on the effect, that is, on what the activity led to. In contrast, low-scoring case studies tended to link backwards, foregrounding the research activity (e.g. “(findings) of the research”). Moreover, they were more likely to include ambiguous or uncertain phrases like “a number of” (implying that it is not known how many) and “an impact on” (implying that the nature of the impact is not known).
  • High-scoring case studies made causal relationships more explicit (above average when compared to general English texts, where low-scoring case studies are below average), which makes a text easier to process.

For REF2021:

  • In Section 2, describe the key findings from the underpinning research that pertain to your impact. Only include essential contextual material, and avoid unnecessary detail on methods or other findings that were not integral to the impact 

2. Articulate how specific groups have benefited and provide evidence of significance and reach 

In REF2014:

  • 84% of high-scoring cases articulated significant and far-reaching benefits, compared to 32% of low-scoring cases, which typically focused on pathway. 
  • High-scoring impact case studies contained more phrases that specified reach (e.g. “in England and”, “in the US”), compared to low-scoring case studies that used the more generic term “international”, leaving the reader in doubt about the actual reach. They also included more phrases implicitly indicating the significance of the impact (e.g. “the government’s” or “to the House of Commons”), compared to low-scoring cases that emphasized pathways (e.g. “the event”, “has been disseminated”).
  • Phrases containing variations on the word “dissemination” were more common in low-scoring case studies. These case studies focused more on pathways than impacts.

For REF2021:

  • Systematically explain why each impact is significant and far-reaching, referring to evidence.
  • Dissemination is not impact: even if you have impressive numbers of reads, downloads, views or listens, how do you know if anyone learned anything from it, benefited, or did anything different as a result? Keep asking “what happened next” until you find the benefit. 
  • For policy impacts, where possible include evidence that the policy was implemented, enforced and worked on the ground.

3. Make your narrative easy to understand 

In REF2014:

  • High-scoring impact case studies scored more highly on a readability measure correlated with reading speed because they included shorter, less complex sentences. The Flesch Reading Ease score, out of 100, was 30.0 on average for 4* and 27.5 on average for 1*/2*. While this is a significant difference (p<0.01), the scores indicate that case studies are generally of “college-graduate” difficulty. 

For REF2021:

  • Aim to write for an interested and educated non-specialist audience. Would your colleague down the corridor, working in a completely different corner of your Unit of Assessment, understand your text on first (and fast!) reading?
  • Break up the text to make it easier to read, using sub-headings and other structural devices 
  • Put evidence in context to explain its significance, e.g. rather than just citing visitor numbers, say if these was the largest number of visitors in any week that year, or explain who visited and why they are significant (perhaps they are from groups who don’t typically visit museums). A small number may be significant in context, but you will need to create the argument to justify this interpretation of your evidence.

Our analysis suggests that impact case studies are a distinctive form of writing. The most successful case studies were written very differently to a normal academic style. This indicates that being the world expert on a topic, writing successfully for academic or even professional audiences, does not make you an expert in writing an impact case study. It is important to take advice on how to evidence and communicate your impacts, and if you don’t have as much support as you would like, we hope our work will give you the pointers you need to get the credit you deserve for your impact.

 


This post draws on the authors’ co-authored paper: Writing impact case studies: a comparative study of high-scoring and low-scoring case studies from REF2014, published in Palgrave Communications.

Note: This article gives the views of the author, and not the position of the LSE Impact Blog, nor of the London School of Economics. Please review our comments policy if you have any concerns on posting a comment below.

Image Credit: MJ S via Unsplash. 


 

Print Friendly, PDF & Email

About the author

Bella Reichard

Bella Reichard is a PhD researcher at Newcastle University, where she investigates the language of impact case studies, comparing high- and low-scoring case studies.

Mark Reed

Mark Reed is a professor of socio-technical innovation in the School of Natural and Environmental Sciences at Newcastle University.

Jennifer Chubb

Jennifer Chubb is a postdoctoral researcher at the University of York. Jenn’s research career to date has focused on the philosophy and politics of research, identities and research cultures, research impact and science policy and advice. Jenn has a particular interest in the impact and ethical implications of emerging technologies. Her current research focuses on the impact of Artificial Intelligence and related digital technologies.

Ged Hall

Ged Hall provides training, development and coaching in impact for academics at the University of Leeds.

Lucy Jowett

Lucy Jowett is research impact manager at Northumbria University, where she works to improve research impact outcomes.

Alisha Peart

Alisha Peart is research impact manager at Northumbria University, where she is developing and delivering strategies to embed impact in the research culture, and improve research impact outcomes.

Andrea Whittle

Andrea Whittle is a professor of management and organisation studies at Newcastle University Business School.

Posted In: Academic writing | REF2014 | REF2021

Leave a Reply

Your email address will not be published. Required fields are marked *

This work by LSE Impact of Social Sciences blog is licensed under a Creative Commons Attribution 3.0 Unported.