Commenting on the recent review of Australia’s ERA and EIA research assessment exercises, Ksenia Sawczak argues that they lack a clearly defined purpose, or return on investment for Australian universities. In a climate of declining trust in the Australian Research Council, together with a confused idea about how research should be funded, she suggests the assessment regime itself is at a critical point of juncture.

In 2023 and 2024, Australia is set to run its two research assessment exercises – Excellence in Research for Australia and Engagement and Impact Assessment. It will be the fifth iteration of ERA, but only the second of EIA. In the lead up, the Australian Research Council – as the administering body -has recently completed a review of these exercises to “ensure the national research assessments address Australia’s future needs”.  What these needs are is a matter of speculation.

The review started with a consultation paper comprising over 100 questions and culminated with a report released by the Review Advisory Committee and the ARC’s Action Plan in response. This report has been described as a missed opportunity and there are a number of reasons why it does not inspire confidence. To start with, the report is laden with vague statements and grandiose assertions that do not bring to bear the benefits these exercises are meant to provide.

This is evident in objectives that pay homage to some old rhetoric, such as assessment exercises providing “a rich and robust source of information on university research excellence and activity to inform and support the needs of university, industry, government and community stakeholders”. Such rehash is hard to swallow given that so far evidence of how assessment outcomes are utilised in any meaningful manner has not been presented by the ARC, as I have discussed elsewhere. The lack of evidence also raises the important question of return on investment given the costs of conducting these exercises. Finally, the 50 plus references in the report to notions of “informing” as a metaphor for utility are more than overkill. They show a lack of thinking in identifying a concrete purpose for the assessment of research.

 Australia has developed highly sophisticated exercises that are unsophisticated in their application

The ARC’s Action Plan in turn does not elucidate on the benefits of these complex exercises. This is the great paradox: Australia has developed highly sophisticated exercises that are unsophisticated in their application, meaning they ultimately do not rise above being mere checks of university research activity.

After 4 rounds of ERA over 11 years, is this really the best we can come up with? The UK made the purpose of Research Excellence Framework as a mechanism for allocating research funding on the basis of performance clear from the very outset, as did New Zealand with their Performance-Based Research Fund. Yet Australia – perhaps the trailblazer in designing assessment exercises – is still struggling with the basic matter of utility. This is quite an oddity when one considers history.

When an evaluation of research and impact was first put to the table in the form of the Research Quality Framework, the Government of the day clearly articulated the role that it would play in distributing research funding to universities on the basis of their performance. The university sector was supportive of this, but expressed concern about the costs of running this exercise and stipulated that the Government needs to meet these in full – including those incurred by universities.

Following a change of government in 2006, the RQF was abandoned and replaced with ERA – an assessment exercise that would focus solely on the quality of research undertaken in universities. Somehow, the matter of performance-based funding failed to be included as an objective and the university sector’s request that they not be lumped with the costs of participation was overlooked. Yet, the matter of performance-based funding was never entirely off the table. Some experimentation occurred, with ERA outcomes briefly used for informing one element of the doomed Sustainable Research Excellence scheme. However, the scheme didn’t last long, and the matter has not been revisited since then.

So why has Australia not been able to devise a way forward that harmonises assessment with performance-based funding? And, more oddly, why do universities participate in these exercises when they bear the cost of preparing submissions but get nothing in return?

Regarding the first matter, the answer may be very simple and explain why the ERA/EIA Review Committee’s Terms of Reference did not include investigating the utility of ERA/EIA for performance. Perhaps we don’t know how research funding for universities should be allocated and the ARC is not prepared to address the matter.

The notion of rewarding research excellence or research impact – which our assessment exercises strive to identify – has rarely come into the conversation.

Australia’s history of formulae used for the allocation of government research funding – the Research Block Grant scheme – is quite interesting. Over the years, changes to the formulae have typically occurred as a consequence of Government’s focus on a particular issue. When the concern was to raise research productivity, a points per publication system was put in place, which stuck until 2016. When the concern was to provide funding to help cover the unmet costs of research, greater weighting was put on Category 1 funding (Australian competitive grants). And, in more recent times, when the Government sought to incentivise greater university-industry collaboration, the weighting formula changed in favour of Category 2 and 3 funding. Thus, government funding has bounced round from being a tool for incentivising behaviours, rewarding desired behaviours, and covering funding short falls. The notion of rewarding research excellence or research impact – which our assessment exercises strive to identify – has rarely come into the conversation. To further complicate the issue, the National Science Research Priorities – once called National Research Priorities before humanities and social sciences were sidelined – play no role whatsoever in defining where research monies should be invested.

Regarding the matter of universities’ continued participation in research assessment exercises, this could be a bizarre form of FOMO. The ARC, with its significant power as a funding agency, is able to maintain a compliant sector, as universities are loath to be at odds with what may or may not transpire. But it is also an increasingly angry sector that is on the verge of losing faith in the ARC. This stems from the ARC’s recent rulings on fellowship applications that were deemed ineligible for breaching new rules – which are at odds with international practices – about citing preprints. One can’t help wondering at what point anger may start to affect compliance.

As things stand, the Government is committed to pushing ahead with the assessment exercises. Adjustments will be made to rating scales and methodologies, as per the Review Committee’s recommendations, and hopefully there will be improvements in streamlining and automation. But, one thing will certainly not change. And that is that, as far as purposes goes, the assessment exercises will continue to be nothing more than audits.

Note: This article gives the views of the author, and not the position of the LSE Impact Blog, nor of the London School of Economics. Please review our comments policy if you have any concerns on posting a comment below

Image Credit: Adapted from Denise Jans, via Unsplash.