Reflecting on the diversity of global research assessment reform movements, Karen Stroobants calls for reform that is ‘as aligned as possible, but as diverse as necessary’. Discussing how different motivations and solutions reflect the contextual nature of research assessment systems, she suggests more efforts are needed to frame and align reform movements around responsible research assessment principles. 

This is the third in a three-part series of discussions together with Noemie Aubert-Bonn, Claire Fraser, Elizabeth Gadd and Haley Hazlett on Unanswered questions in research assessment, you can read the rest of the series here.

​​The reasons that individuals and organisations engage in research assessment reform are diverse. This leads to a diversity of approaches that may or may not be effective outside their local context.

What makes sense in Europe might not in North America. Organisations at the outset of their assessment reform journeys might think differently about their approach compared to those who have been working on them for longer. This presents a challenge to a global movement to improve research assessment processes, as solutions, and even the reasons to introduce them, are heterogeneous.

A myriad of motivations

When examining the rationale for individuals and organisations to take part in research assessment reform, multiple sometimes conflicting reasons quickly become apparent:

Take for instance reforming organisations’ calls to action. In 2013, DORA stood out for its call to do away with journal-based metrics, specifically the Journal Impact Factor, in researcher assessment. The primary reason being that research outputs should be assessed on their own merits rather than journal performance.

Equally frustrated with the over emphasis on the Journal Impact Factor for researcher assessment, the Latin American Forum on Research Assessment (FOLEC) calls attention to the geographic, linguistic, and disciplinary limitations of databases, such as WoS and SCOPUS, that are used to extract citation indicators.

The Agreement on Reforming Research Assessment (ARRA), that sets out the guiding principles and commitments of the Coalition for Advancing Research Assessment (CoARA),explicitly mentions as underlying reasons for reform: to allow research assessment to support positive research cultures; to ensure assessment practices stay relevant as research processes and the expectations of research evolve; and to live up to the increasing demands placed on research by the many societal, environmental, democratic, and economic challenges we face.

Established in 2021, the Global Research Council (GRC)Responsible Research Assessment Working Group acknowledges that global approaches to responsible research assessment are pivotal to high-quality research and impact but they must be cognisant of local context, culture, and language.  

Divergent motivations lead to heterogeneous solutions

While divergence in motivations is welcome, it inevitably results in a variety of views on what new criteria, tools and processes should be prioritised. This is illustrated in part by the emergence of specific groups such as the former European Commission metrics working group focusing on assessment of open science practices, the Pew Charitable Trust’s Transforming Evidence Funders Network advocating for the value of public engagement; as well as the recognition of peer review or applied/practice-based research being put forward within CoARA. It has also led to active discussions around the extent to which certain aspects such as ‘equality, diversity & inclusion’ are accepted as central themes of the reform, and the spectrum of views on the right balance between qualitative and quantitative approaches and the attention these deserve in reform activities. 

While divergence in motivations is welcome, it inevitably results in a variety of views on what new criteria, tools and processes should be prioritised.

As different actors with specific interests in the reform arena swiftly move forward with development of new criteria, tools and processes serving their own priority area, there is growing need for consideration as to how their proposals fit into global reform efforts.

Convergence where it matters

In the midst of the complexity of views on why we are doing this and what new criteria, tools and processes should look like as a result, we should not lose sight of the fact that organisations charged with implementing new policies and processes will lose their motivation to contribute if their goals are unlikely to be achieved. The GRC Responsible Research Assessment Working Group’s eleven dimensions of responsible research assessment can be useful to frame individual and organisational priorities among the range of aspects considered across the community. While the SPACE rubric and workbook created by DORA and the CoARA action plans and self-assessments offer ways research organisations can chart their progress and plan next steps.

Structured narratives are another promising alignment tool as they provide space to capture wider contributions to research. For example, the NIH Biosketch and Resume for Research and Innovation focus on contributions made from individuals, whereas the INORMS More than Our Rank initiative uses structured narratives to describe the achievements of an organisation. Structured narratives avoid the ‘box-ticking’ risk inherently associated with numerical indicators. Of course, narrative approaches are not without their challenges and it is important that their implementation considers appropriateness at different levels of granularity and goes hand in hand with evaluation of their effectiveness.       

As the number of responsible research assessment initiatives continues to grow, the challenge of remaining aligned to maximise progress becomes more important.

As the number of responsible research assessment initiatives continues to grow, the challenge of remaining aligned to maximise progress becomes more important. These initiatives have an important role to play in developing mechanisms to ensure a certain level of coherence, and as we argued in the second piece in this series, agencies need to be given the freedom to learn from new research assessment practices. Over the past years, many of these initiatives have converged around the ambition to recognise more diverse contributions while being cognisant of local context, culture, and language. They continue to engage in discussions to further refine a common understanding amidst the layers of divergence inherent to the reform exercise. This requires thought and care to ensure the right balance between strength in alignment and strength in context-specific variation.

Just as the open research movement has adopted the phrase, ‘as open as possible, but as closed as necessary’, perhaps the research reform movement needs to adopt the slogan, ‘as aligned as possible, but as diverse as necessary’?

DORA’s communities of practice, FOLEC’s working groups and CoARA’s working group co-chairs meetings and national chapter fora for exchange, to name a few, offer productive routes to enable the community to work towards this goal. However, these will likely need to be complemented by more targeted efforts such as a regular global research assessment conference or summit supported in design and analysis of outcomes by experts in research assessment.

Achieving convergence ultimately requires an effort from all involved to understand the range of reasons and approaches and a level of compromise from personal motivations. We must keep sight of the ultimate goal: to improve research and research environments – by accomplishing better rather than mythically ‘perfect’ forms of research assessment.

We would like to acknowledge the thought-provoking insights of Anna Hatch to the discussions leading up to the creation of these posts.

The content generated on this blog is for information purposes only. This Article gives the views and opinions of the authors and does not reflect the views and opinions of the Impact of Social Science blog (the blog), nor of the London School of Economics and Political Science. Please review our comments policy if you have any concerns on posting a comment below.

Image Credit: Dedraw Studio on Shutterstock.