The Leadership Foundation for Higher Education’s Director of Research, Professor Fiona Ross CBE commissioned Dr Elizabeth Morrow to mine the 2014 REF impact case studies to learn more about leadership, governance and management (LGM) research. The case study data suggests new possibilities for inter-professional collaboration, and also elucidates different types of impact, which may not necessarily be sequential. Furthermore, given the huge planned changes in UK higher education it is essential that LGM research is valued and used to inform future directions for the sector.

In the wake of the Higher Education White Paper, universities face some considerable challenges to show and improve their educational, research, social and economic impact. Whilst the term ‘impact’ is bandied about and we think we know what it means, there is little consensus about how impact is achieved, what kind of organisational conditions support and nurture it, or what success looks like.

In a shifting higher education landscape, universities need steady anchors and clear compasses to orientate themselves towards success – whether this is determined by impact on student satisfaction, learning analytics, graduates going into graduate-level jobs, world university rankings, social impact, all of these things, or something else – impact requires direction and it requires effort to get there. This is where leadership, governance and management research can help universities.

To stake-out some of the impact territory, the Leadership Foundation for Higher Education commissioned a study of impact case studies from leadership, governance and management (LGM) research submitted to the 2014 Research Excellence Framework (REF) assessment. A searchable database of over 6,000 case studies is freely available online. The study used a combination of text mining and qualitative research methods to analyse a sample of 1,309 LGM case studies. We explored how they had led to change, made a difference or had impact on the higher education sector and to other diverse areas too, such as the police, armed forces, business and industry, health and social care, tourism and sports science.

The results provide a view of impact from LGM research across the UK. They show that there is substantial LGM research going on in academia, which has largely been obscured from view until now because it cross-cuts disciplinary boundaries. A substantial proportion of the 154 UK universities that submitted to the 2014 REF (86%, n=131 HEIs) returned LGM case studies. The average per HEI was 8.5, with some submitting over 30 LGM case studies.

Unsurprisingly the majority of LGM case studies were found to be within units of assessment for Business and management studies (n=243), Education (n=108), Politics and international studies (n=94). However over two-fifths of LGM case studies were returned by other subject areas not often perceived as contributing to LGM research, including: Allied health professions, dentistry, nursing and pharmacy (n=85), Social work and social policy (n=80), Psychology, psychiatry and neuroscience (n=64), Communication, cultural, media, library & information (n=33), Clinical medicine (n=31), History (n=30), and Computer science and informatics (n=28). This view of LGM research opens up new possibilities for inter-professional collaboration, it also raises questions about the impact of clustering teaching and research themes outside of disciplinary boundaries.

Overall the evidence of LGM research impact is significant in terms of:

• Use of evidence (type I impact) to inform law, government policy, strategy, regulation, standards, guidelines or recommendations for practice, or priorities for research or practice
• Use of research products (type II impact) such as a training programme or course, intervention (programme or initiative), toolkit, model, decision aid, system (operational), support network, technology, measure, benchmark, information resource, research method, participation method, teaching method, visual art, music or fiction
• Effect on individuals (type III impact) such as change in awareness, attitude, understanding (knowledge) or behaviour
• Effect on groups/organisations (type IV impact) including knowledge-transfer (intra-organisation, inter-organisation or network), organisational development (innovation, new systems or structures), or organisational performance (impact on end-users).

The categories above could imply different stages in reaching impact (from type I to type IV impact). However, the case study data shows that these types of impact are not necessarily sequential and research use may not always lead to measurable effects. Analysis of outcomes from these 1,309 case studies using text mining shows 45% (n=2,865) of outcomes are type II (research products), and 34% (n=2,178) are type I (use of evidence). Far fewer outcomes fall into categories of effect on individuals (16%, n=989), or effect on groups/organisations (5%, n=297).

Owing to the fact that REF 2014 focused on impact “beyond academia” there is only limited evidence of the impact of LGM research on the higher education sector itself – a missed opportunity for the sector to inform its own development. Analysis of a subsample of 20 selected LGM case studies shows that impact of LGM research on universities included: improvements in leadership skills and behaviours, equality and diversity in leadership roles, coordination and leadership capacity development, management development, change management, organisational improvement, and new LGM research methods. Given the huge planned changes in UK higher education it is essential that LGM research is valued and used within the sector.

While the findings of this analysis point to a number of outcomes for LGM research at the end-user level in higher education (student experience, student engagement, student retention, student learning or attainment, staff and student well-being, staff skills or knowledge, and staff commitment or employee engagement) it is not always going to be possible to isolate the effect of research on individuals or groups/organisations at this level. It is difficult to show direct causal links between LGM research and effect at the end-user level because research impact is affected by the interfaces between research/research users and researchers/end-users.

On average, the reported time to impact of LGM research in higher education contexts was 7 years. The earliest impact or “quick wins” (mean average 5 years) tended to be use of evidence (type I impact). It generally took longer (averaging 8.3 years but with a notable peak at 6 years) to develop research products, such as training programmes or interventions, and to see the effects of research on individuals (8 years), groups or organisations (7.3 years).

Analysis of spread in higher education contexts shows that the first reported place of impact was most often at a national level (e.g. impact on national policy or programmes) with subsequent spread of impact towards the local (university) level. These findings suggest that the main route that impact spreads between universities is by the influence of national bodies and organisations.

The emergent and dynamic process of achieving impact challenges notions of linearity in impact planning and assessment. We found eight interrelated processes of impact generation through which impact may be cultivated: researcher impact skills, contextual leverage, stakeholder engagement, public/user involvement, designing research for impact, mechanisms for exchange, developing impactful outputs, and implementing and evaluating outcomes. In the report these processes are developed in an Adaptive Systems Framework for Advancing Research (AS-FAR) to enable researchers to plan, predict and assess LGM research impact, as far as possible, within a complex and adapting system.

There are also lessons here for researchers about reporting on research impact. Case studies that made the most convincing claims to impact show causality, use measures and indicators, address issues of attribution and contribution, emphasise progression or spread of impact, show systematic capture of impact information, and present a tailored account using an active authorship style.

There is still much to learn about the effectiveness of different routes to impact in different contexts. However, there is good evidence in the case study data about mechanisms for stimulating interest in research, ways of keying into the interests of beneficiaries, mechanisms for spreading knowledge, and ways to build knowledge. The data suggests that the contribution of public/user involvement to research impact includes research that is more relevant and able to address public/user views, greater spread of impact of research through public/user groups and networks, improved public/user capacity for involvement in research, and enhanced research practices and researcher skills to work with the public/users.

The implications for aligning skills development with what works in impact are far reaching. Senior executives, managers, researchers, support teams, and so on, can consider how they position themselves and assess whether they collectively have the right skills for impact. So far skills assessment and development has been approached as an issue for individual staff development. Yet what we do know about successful impact (and also leadership, governance and management) is that it is created and sustained through the interactions of many, not the individual actions of a few.

Impact is changing the landscape of higher education. It is important not to drift away from issues that fall outside the core remits of the new funding and assessment arrangements. Issues such as the place of multi- and inter-disciplinary endeavours, the position of research and teaching within and between universities, and the impact on those people directly involved in delivering it. To avoid repeating the missed opportunities of REF 2014, it is really important that universities, as a knowledge industry, are encouraged to ask serious questions about organisational effectiveness, and are rewarded for building knowledge that informs the sector.

Note: This blog is based on the report ‘The Impact of Higher Education Leadership, Management and Governance Research: Mining the 2014 Research Excellence Framework Impact Case Studies’ April 2016. A summary report and the full report are available online to members of the Leadership Foundation for Higher Education.

A short video about the research can be viewed on YouTube at:

https://www.youtube.com/watch?v=A8-vGAAbK4I

Note: This article gives the views of the author, and not the position of the LSE Impact blog, nor of the London School of Economics. Please review our Comments Policy if you have any concerns on posting a comment below.

About the Authors

Professor Fiona Ross CBE (@LF4HE) is Director of Research at the Leadership Foundation for Higher Education.

Dr Elizabeth M Morrow (@elizabethmmorr1) is an independent researcher based in Northern Ireland. She is interested in the power and politics of research practice.