Over the last couple of decades there has been an international push around the assessment of the wider societal impact of research. Kate Williams and Jonathan Grant document the evolution of research impact assessment in the UK and Australia, and how policies in the two countries have been seemingly interdependent, a back-and-forth process developed through international learning. Continued political commitment to impact assessment is likely in both countries, with debate centred around reducing the costs and burden through the use of impact metrics.
In recent years, the university has undergone a shift from its earliest function as the primary place of learning and scholarship, to its contemporary role as service-provider in a complex commercial landscape. Within this context, there has been a recent international push around the assessment of the wider societal impact of research. Based on the initial impression that approaches in Australia and the UK have been synergistic, we document the evolution of research impact assessment in the two countries. In our research paper, we undertook a systematic analysis of public policy documents, newspaper commentary, and the academic literature in both countries. We identified four phases of policy development, detailed below (and summarised in the figure from our paper). We argue that research impact assessment policies in the two countries have been interdependent.
Phase 1: Australia develops but abandons impact assessment
The genesis of impact assessment in Australia can be traced back to political commitments made in 2001 to strengthen links between industry and universities. These commitments were formalised in 2005, with the establishment of an expert advisory group to provide guidance on the development of the proposed Research Quality Framework (RQF), focused on both the quality of research and its broader impact or use.
In January 2006, a newly formed Working Group on Research Impact was asked to make recommendations on an optimal methodology for assessing impact. Instead of an indicator approach, it was recommended that assessment should rely on evidence-based impact statements containing both qualitative and quantitative information. The impact component was expected to make up at least 10% of a formula for allocating funding to universities. Assessment was scheduled to start in July 2008, with funding implementation in 2009.
Despite these plans, development of the RQF ended with a change of government in December 2007, with the centre-left Rudd Labour Government coming into power. Shortly following the election, it was announced the RQF would not proceed because it was badly designed, expensive, and based on inadequate impact measures.
Phase 2: the UK “steals” the Australian approach
Early interest in research impact within the UK is evident in policy papers from 1993 advocating that the benefits of scientific research should accrue to society at large. Yet, genuine commitment came in March 2006 with the plan to replace the Research Assessment Exercise (RAE) with a cheaper and more streamlined system based on quantitative measures. This proposal faced widespread opposition. A consultation on the replacement of the RAE highlighted concerns that a metrics approach would be unable to gain an adequate picture of user value and impact, and indicated a preference for peer review over metrics.
The proposal for the revised framework (now called the Research Excellence Framework or REF) included a new strategic aim that research should contribute to economic prosperity, national wellbeing, and knowledge dissemination. In February 2009, a commissioned review of international impact assessment approaches found that the Australian RQF provided the best foundation for developing an impact framework. During 2010, the UK developed a preliminary methodology that adopted the RQF’s use of a case study approach, with additional criteria. However, the introduction of an impact measure to the REF was controversial. Politicians vacillated around the issue, and critics stressed the explicit steering of researchers and universities.
In March 2011, the incorporation of impact in the REF was confirmed, accounting for 20% of the total assessment (research outputs accounted for 65% and research environment for 15%). In 2013, 154 higher education institutions submitted 6,975 impact case studies. Panels evaluated submissions throughout 2014, and results were published later that year.
Phase 3: Australia puts impact assessment back on the agenda
Despite the newly elected Australian Labour Government abandoning impact assessment in December 2007, it returned to the agenda four years later when a research impact mechanism was recommended in a review of publicly funded research. A feasibility study on possible ways of assessing the benefits of publicly funded research was conducted in 2012, and a trial impact assessment was simultaneously undertaken by select universities.
In December 2015, the national Engagement and Impact (EI) assessment was announced, in order to examine the translation of university research into economic, social, and other benefits and incentivise greater collaboration with industry and end-users. However, a consultation on the EI highlighted widespread criticism relating to the lack of underpinning conceptual and methodological work compared to that done before the UK REF.
An EI pilot took place in the first half of 2017, which tested a mixed-methods approach involving a matrix of metric indicators with an accompanying narrative and impact case studies. This approach, due to be implemented nationwide in 2018, is an innovation from the UK’s REF. First, it differentiates “engagement” from “impact”, which potentially creates confusion in terminologies and suggests a hierarchy of types of impact, which the UK seeks to avoid. Second, it uses different processes for assessing engagement and impact. Unlike the UK model, the Australian system uses metric indicators, supplemented with narrative case studies. The divergence of the two national systems makes possible further separation in future.
Phase 4: the UK demonstrates ongoing commitment to impact
Following REF 2014, the UK commissioned a number of evaluations to inform future policy development. A two-phased evaluation outlined a number of advantages of, and relatively high confidence in, the increased focus on impact, but a corresponding and significant burden on HEIs and a number of unresolved uncertainties and concerns. One key conclusion from REF 2014 evaluations was that metric impact indicators are not sufficiently developed to be used to make funding decisions, reaffirming the use of narrative case studies.
In 2016, Lord Stern led an independent review of the REF, which identified a number of issues around impact assessment. The report recommended a significant broadening and deepening of the notion of impact, and that the total weighting for impact comprise no less than 20% in the next exercise.
Following a consultation, initial decisions on the next REF were published in September 2017, which included an increase of the impact weighting from 20% to 25%. The increase was couched in the language of supporting industrial strategy, mirroring the Australian emphasis on business engagement and knowledge exchange. Several challenges remain, yet impact assessment has broadly been considered a success by the Government and funding councils, suggesting ongoing commitment to impact in future policy development.
Although the UK is widely considered the leader in research evaluation, it is clear the evolution of impact assessment has been a back-and-forth process between Australia and the UK. Each country’s policy development has been developed through international learning. We are likely to see a continued political commitment to impact assessment in both countries, with the core debate centred around reducing the costs and burden through the use of impact metrics.
Australia is more likely to take the lead on the development of lower-cost, metrics-based systems, because their assessment is not associated with direct financial incentives. The costs of assessing research impact in the UK can be justified given the high level of funding that follows, but this justification is more difficult in Australia. However, given the iterative policy development to date, this metrics-based approach may subsequently be picked up by the UK. Thus, the two countries are likely to continue their co-dependence in developing policy and procedures to assess research impact.
This blog post is based on the authors’ article “A comparative review of how the policy and procedures to assess research impact evolved in Australia and the UK” published in Research Evaluation (DOI: 10.1093/reseval/rvx042).
Note: This article gives the views of the authors, and not the position of the LSE Impact Blog, nor of the London School of Economics. Please review our comments policy if you have any concerns on posting a comment below.
About the authors
Kate Williams is an Economic and Social Research Council Future Research Leaders Fellow in the Department of Sociology and a Research Fellow at Lucy Cavendish College. She is also currently a Postdoctoral Fellow at the Weatherhead Centre, Harvard University. Her research focuses on the production, use and evaluation of policy knowledge, particularly on emerging digital methods of impact assessment. Originally from Australia, Kate moved to Cambridge in 2012 to complete her PhD.
Jonathan Grant is the Vice-President/Vice-Principal (Service) at King’s College London. Service has been identified as one of five strategic priorities in King’s Strategic Vision 2029, alongside Education, Research, London, and International. Service encapsulates King’s commitment to society beyond education and research and will contribute to the realisation of its vision to make the world a better place. In addition, Jonathan is Professor of Public Policy in the Policy Institute at King’s. Jonathan was Director of the Policy Institute between February 2014 and 2017.