Noemie Aubert Bonn and Haley Hazlett discuss the importance of applying scholarly standards to research assessment reform efforts and the value of creating space for experimentation, transparency, and openness in research assessment initiatives.

This is the second of a three-part series written with Claire Fraser, Elizabeth Gadd, and Karen Stroobants, on Unanswered questions in research assessment, you can read the rest of the series here.

Good research requires much of research-active and research-enabling communities, and the academic organisations that support them. It depends on a body of knowledge to create well-informed hypotheses, rigorous testing of those hypotheses, and an honest commitment to reproducibility and transparency in research design, data analyses, and reporting. Importantly, it calls on communities to become intimately familiar with negative results and errors (despite best efforts to avoid them).

It is therefore surprising that one of the most common criticisms levelled at research assessment is that it does not adhere to the same standards of rigour we apply to academic research. 

Experimentation and iteration are necessary for reform

The approaches and methods used to implement assessment reform are as varied as the research ecosystem itself. The number of research-performing organisations and funding bodies that are implementing and experimenting with new responsible research assessment methods has been growing and include organisations like: FNR,CIHR, MBIE and NHMRC, Leiden University, the Open University, the University of Calgary, the over 3,000 organisations who have signed DORA, the 600 organisations who joined CoARA, and likely many more.

By virtue of their newness, evidence for new research assessment methods is relatively scarce. 

These organisations are working hard to improve their systems of evaluation to better recognise the diverse and important work of researchers. This could include approaches as varied as using narrative CVs, creating new promotion and career tracks, providing greater transparency in application requirements, reviewing approaches to reviewer panel composition and training, or even creating more specific criteria to assess team science and transdisciplinary research. 

As exciting and inspiring as the flurry of new methods, tools, and resources is, important considerations remain around the rigour, long-term impact of, and risk aversion towards, reforming research assessment. By virtue of their newness, evidence for new research assessment methods is relatively scarce. This limits the space for reform projects to have the freedom to experiment, iterate, and learn from the interventions they implement. Ultimately a lack of freedom to experiment stymies the evidence base needed for creating better research assessment systems.

The need for iteration 

To secure the future of research assessment and reduce the risks associated with new assessment methods, changes in research assessment need to be thoroughly tested, piloted, and documented. Everyone involved has a responsibility to document efforts and to rigorously, openly, and transparently report all results. Efforts to improve research assessment are, after all, still largely a form of research. As such they should abide by the standards we expect from research, in particular the ability to learn and adapt to negative results. Without this willingness to trial, study, document, share, and celebrate lessons learned from new research assessment practices, any reform risks losing momentum. 

Efforts to improve research assessment are, after all, still largely a form of research. As such they should abide by the standards we expect from research

In essence, responsible research assessment advocates and policymakers need to embody the approaches of the community they are trying to serve: thoughtful experimentation and acceptance of incremental change, trial and error. This concept can also be applied to international initiatives that advocate on a global scale.

Global change is a systems-within-systems problem. Every country and region has its own requirements, capacity for change, and processes. And while change on a global scale requires collective action towards a shared goal (e.g., to better recognise the qualities of research and researchers that we care about), it also requires that advocates not fall into the trap of mistaking homogeneity for success. Creating space for and accepting diverse, context-specific reform efforts is critical for continued buy-in across the world, even as the world moves together towards this shared aim. This need for a better understanding of and recognition of context specific research form is an issue we seek to emphasise throughout this series of posts.

Allow organisations to be open about what they learn

Institutions and any other organisations and initiatives aiming to change research assessment should be given permission to be as open, transparent, and honest about their results as researchers are expected to be. Several initiatives already exist to capture such experiences, like the Latin American Forum on Research Assessment (FOLEC), the Declaration on Research Assessment (DORA),and the Coalition for Advancing Research Assessment (CoARA). Project TARA and the DORA case studies and communities of practice, alongside the mutual learning opportunities within communities such as CoARA’s national chapters and working groups, the GlobalResearch Council Working Group on Responsible Research Assessment, Meta Research Open Review (MetaRoR), and the global observatory of responsible research assessment (AGORRA) are prime venues where experiences, successes and lessons can, and should, be shared.

the research community also shares a responsibility to approach and listen with an open mind.

In turn, the research community also shares a responsibility to approach and listen with an open mind. Reforming research assessment is a collective effort to make research and research careers better, and all involved should understand the necessity for this phase where learning occurs. It is widely recognised that assessment reforms may be particularly challenging for early career researchers who find themselves in this transition period with changing requirements and expectations. The community, assessors, and changemakers should be sensitive to these challenges, and proceed with kindness, patience, openness, and generosity so that we support each other during this transition period. Ultimately, it is only through a shared and collaborative mindset that we can be free to experiment, learn, and enable positive and lasting change. 

We would like to acknowledge the thought-provoking insights of Anna Hatch to the discussions leading up to the creation of these posts.

The content generated on this blog is for information purposes only. This Article gives the views and opinions of the authors and does not reflect the views and opinions of the Impact of Social Science blog (the blog), nor of the London School of Economics and Political Science. Please review our comments policy if you have any concerns on posting a comment below.

Image Credit: National Cancer Institute on Unsplash.