LSE - Small Logo
LSE - Small Logo

Blog Admin

December 30th, 2017

2017 in review: round-up of our top posts on research evaluation and impact

0 comments

Estimated reading time: 5 minutes

Blog Admin

December 30th, 2017

2017 in review: round-up of our top posts on research evaluation and impact

0 comments

Estimated reading time: 5 minutes


Increasing REF’s impact weighting could offer incentive for institutions to address societal, economic and global challenges

matthew-guestChallenges posed by events such as Brexit highlight the importance of excellent research programmes. Moreover, they represent a broader context in which the next Research Excellence Framework must consider “impact”. But do current REF proposals go far enough towards doing this? Matthew Guest argues that there is not enough of an incentive for institutions to address heightened societal, economic, and global challenges because proposals do not directly link into the wider national agenda. Increasing the overall weighting for impact might go some way towards rectifying this problem.

Does high-impact research come at the expense of quality? An automated analysis of the REF impact landscape

As part of the 2014 Research Excellence Framework, the impact of research was assessed for the first time. But how effective was the impact category in capturing the many diverse forms of interaction between academia and society? Were certain interpretations of impact more highly rewarded than others? And does high-impact research come at the expense of quality? Martin Zaltz Austwick and his research team used a text mining technique to analyse impact case study submissions and address some of these questions.

The REF’s focus on linear and direct impact is problematic and silences certain types of research

In the last Research Excellence Framework (REF), the new element of research impact was understood in very linear and direct terms. Aoileann Ní MhurchúLaura McLeodStephanie Collins and Gabriel Siles-Brügge consider how accepted definitions of impact may have had the effect of silencing certain types of research. Research and impact should be seen as a two-way street, where academics engage with their research subjects as part of a process of co-production. Moreover, impact must be thought of as a collective endeavour that captures the broader social and cultural benefits of academic work. The Stern Review recommendations are to be welcomed but whether and how they are adopted remains to be seen.

The 2014 REF results show only a very weak relationship between excellence in research and achieving societal impact

Results of the 2014 Research Excellence Framework have, in some quarters, been interpreted as evidence of a direct relationship between the quality of scientific outputs and the degree of societal impact generated by researchers. However, such an interpretation, allied to definitions of impact such as that used by Research Councils UK, arguably promotes a stronger reading of the REF results whereby only excellent science leads to high-quality impacts. Richard Woolley and Nicolas Robinson-Garcia looked again at the relationship between the 4* components of evaluated scientific outputs and impacts and found a great deal of variation across all panels. It may be that it is this diversity and heterogeneity that constitutes the strength and vibrancy of UK university research.

Credit for research outputs should go to the originating institution but with a transitional arrangement for this REF cycle

One of the most contentious aspects of the Stern review of the 2014 REF was the recommendation that research outputs should not be portable in future exercises. The subsequent consultation revealed a significant minority to be in support of this, echoing Stern’s concerns that current rules distort investment incentives and encourage rent-seeking. However, a majority opposed this recommendation as stifling of researcher mobility, with many also highlighting the disruption caused by a mid-cycle change. David Sweeney explains that the Stern recommendation will be implemented but that one of two proposed transitional arrangements should also be set in place for the current cycle.

Involving non-academic partners at all relevant stages of the research process can drive knowledge and understanding

At a time when some audiences and commentators seem intent not simply to resist academic knowledge but to discredit it, the perception of academic researchers as somewhat under siege is perhaps unsurprising. But rather than aggressively reasserting the value of academic expertise, Claire PackmanLouise Rutt and Grace Williams argue for a reconceptualising of the meaning of professional community and a more collaborative approach to research. Professional services staff are often ideally placed to bridge any perceived divide between academic and non-academic stakeholders, and to consider ways to dismantle some of the barriers to effective communication.

Where’s the evidence? Obstacles to impact-gathering and how researchers might be better supported in future

Despite the increased importance of demonstrating impact, it remains a concept many academics feel ill-equipped to measure or evidence. Clare Wilkinson reveals how researchers from a broad range of disciplines think about evidencing impact, what obstacles might stand in their way, and how they might be further supported in future. Knowledge around research impact continues to exist in siloes, with those at early-career stages often less clear about it. Researchers are eager to see best-practice examples and information on the types of evidence that could be collected, but are less receptive to support mechanisms that more obviously add to bureaucracy. Interestingly, academics rarely consider how their own research backgrounds, methodologies, and tools may feature in their impact activities.

Research assessments based on journal rankings systematically marginalise knowledge from certain regions and subjects

Many research evaluation systems continue to take a narrow view of excellence, judging the value of work based on the journal in which it is published. Recent research by Diego ChavarroIsmael Ràfols and colleagues shows how such systems underestimate and prove detrimental to the production of research relevant to important social, economic, and environmental issues. These systems also reflect the biases of journal citation databases which focus heavily on English-language research from the USA and northern and western Europe. Moreover, topics covered by these databases often relate to the interests of industrial stakeholders rather than those of local communities. More inclusive research assessments are needed to overcome the ongoing marginalisation of some peoples, languages, and disciplines and promote engagement rather than elitism.

Institutional support for impact remains at the relatively early stages of embeddedness

An established discourse within UK higher education institutions has developed around the pursuit of “academic excellence with impact”. However, the everyday reality is that, in terms of institutional support and opportunities, the impact element is not as embedded as that of academic excellence. Carlos Galán-Díaz has adapted the EDGE Tool, developed to assess universities’ strategic and practical support for public engagement, to research impact activities and collected data on the degree to which impact is ingrained within the purpose, processes, and people of institutions. Findings do indeed show that institutions are still at the relatively early stages of implementing the mechanisms and governance necessary for impact support and development.

The business of impact: academic reward and incentive cultures continue to stifle relationships between business and management researchers and society

Uncertainties remain over what path the UK economy will take post-Brexit. The government has made clear its plan to leverage the nation’s research strengths to meet the needs of business and society, but new research from Mattia Fosci and Rob Johnson suggests that academic reward and incentive cultures are still hampering efforts to create bridges between business and management researchers and society. Leading universities and business schools need to embrace their social responsibility and work more proactively to create a more inclusive and sustainable economy.

Print Friendly, PDF & Email

About the author

Blog Admin

Posted In: Annual review | Higher education | Impact | REF2014 | REF2021 | Research evaluation | Research policy

Leave a Reply

Your email address will not be published. Required fields are marked *