What are the organisational contexts in which ‘impactful’ research is produced? Following an empirical analysis of a selection of REF2014 impact case studies, Neil Kellard and Martyna Śliwa discuss the links between impact scores and a variety of important contextual factors. In what might be seen as a challenge to the established hierarchy of HEIs, high scores for research publication quality did not necessarily correspond with high scores for impact case studies. However, the under-representation of both early-career and women researchers is a concern future evaluation exercises should seek to address.
For the first time in the history of research assessments in the UK, the Research Excellence Framework 2014 (REF2014) evaluated the impact of research outside academia. Despite much subsequent analysis and discussion of REF2014 results, little attention has been paid to the organisational contexts in which ‘impactful’ research is produced or the researchers who generate it. However, our recent research empirically analyses the impact scores within the Business and Management Unit of Assessment (UoA) to explore these issues, which are of keen interest to university managers and higher education policymakers alike.
Following a comparison between three performance-related clusters of institutions (the top ten, middle ten and bottom ten of the 98 institutions to include impact case studies in their REF submissions), we discuss the links between impact performance and the overall research intensity of institutions, grant income generation, research team size, career stage and gender of academics, and whether impact activity is focused on private or public sector organisations.
Image credit: Inner workings 2013-CMP-7290 by Cynthia Pyun. This work is licensed under a CC BY 2.0 license.
Our analysis gives rise to a number of observations and implications. First, it has demonstrated that high scores for research publication quality did not always correspond with high scores for impact case studies. This meant some higher education institutions not typically considered ‘elite’ universities – for example, Bournemouth University or the University of Brighton – generated excellent impact from their research. Likewise, some ‘older’ universities – for example, London Business School and the University of Surrey – performed relatively poorly on impact. If impact continues to be a part of research evaluation (and its importance may well increase in the future) this could result in a new hierarchy of HEIs thought of as ‘excellent’.
Second, our analysis has shown that the average grade for impact across all 36 UoAs was higher than the equivalent grade for the Business and Management UoA only. Specifically, STEM-discipline units often received higher grades than those in the social sciences, humanities and arts. One possible explanation for this is that the REF2014 rules for impact assessment did not fully capture impact generated by institutions submitting to the Business and Management unit. The construction of knowledge in the social sciences usually follows from debate that takes place amongst a broad community of scholars; scholars who are located across many institutions, and not necessarily working together. Hence, in contrast to the ‘hard’ sciences, it becomes questionable whether drawing a direct link between impact and the work of a single or group of scholars within one particular university is appropriate. As the Stern Review has suggested, it may therefore be helpful to broaden the definition of impactful research so that it is better suited to social science disciplines such as business and management.
Third, within the Business and Management UoA, our analysis has highlighted that successful impact case studies tended to emerge from the collective effort of small teams of predominantly established researchers. Some of these scholars had been employed by the same institution for 20 years or more and had strong track records of obtaining external grant funding. This points to a need for university managers to support the formation and long-term sustainability of collaborating research teams. Such teams are well-placed to secure funding, develop research and engage with external stakeholders over the time periods required to generate excellent impact.
Conversely, the analysis has revealed an under-representation of early-career academics (ECAs) as key researchers behind impact case studies. This category of academics, whose position in the HE environment is already precarious, is clearly less able to produce impact compared to long-serving scholars. University managers may therefore be inclined to neglect the career development of junior staff; a tendency we would counsel against, as the future health of the overall university system relies on the nurturing of ECAs. As a corollary, it is also crucial for HE policymakers to revise the REF impact rules so that they do not unduly disadvantage ECAs.
Fourth, our findings have also demonstrated an under-representation of women amongst the key researchers involved in impact case studies. There might be different reasons behind this. For example, it might be more difficult for women than men to participate in national and international networks both inside and outside academia (e.g. because of the homosociality of academic networks, whereby male academics are more likely to collaborate with other males); or some women might self-exclude from engagement in activity that produces impact on policy and practice (e.g. because of family commitments outside work or the ‘imposter phenomenon’). Whatever the reasons, the current REF impact exercise would appear to reinforce the gender imbalance within the business school context. As with ECAs, we would challenge HE policymakers and university managers to design future impact assessments and organisational processes that would enable greater involvement of women in impact-generating research teams, especially as key researchers.
Finally, the majority of impact case studies were built around work conducted with the public and non-profit sector as opposed to commercial organisations. This may be because the current definition of research impact, which emphasises benefits for “the economy, society, culture, public policy or services, health, the environment or quality of life”, does not unambiguously encourage generation of impact on businesses. Of course, it is also possible that it was more difficult for institutions to evidence research impact on business – perhaps because of confidentiality issues, for example – or that it was more common for public and non-profit sector organisations to commission research from universities. But it might also be that the REF2014 rules precluded some institutions, in particular post-1992 universities, from demonstrating their impact on business and industry. While these rules stipulated that the underpinning research be of “at least ‘two star’ quality”, many business school staff who have traditionally been successful in applied research and knowledge transfer, do not prioritise research output publication. In the case of this category of academics, their engagement with, and influence on, private sector organisations did not count.
This blog post is based on the authors’ article, ‘Business and Management Impact Assessment in REF2014: Analysis and Reflection’, published in the British Journal of Management (DOI: 10.1111/1467-8551.12186).
Note: This article gives the views of the author, and not the position of the LSE Impact Blog, nor of the London School of Economics. Please review our comments policy if you have any concerns on posting a comment below.
About the authors
Neil Kellard is Professor and Head of Finance at the University of Essex. He is also a founder member of the Essex Centre for Financial Econometrics. Neil’s research typically examines the linkages between market efficiency, risk management, derivative markets, growth and development. He has published several papers in international journals such as the British Journal of Management; the Review of Economics and Statistics; the Journal of Development Economics and the Journal of Empirical Finance.
Martyna Śliwa is Professor of Management and Organization Studies at the University of Essex. Her research interests focus on language(s) and power in organizations, professional identities, professional mobility, gender, intersectionality and higher education. Her work has been published in a range of international journals including British Journal of Management; Gender, Work and Organization; Journal of International Business Studies; Management Learning and Organization.