Following the publication of the Agreement on Reforming Research Assessment under the auspices of the European Commission, countries across Europe are reconsidering their research assessment systems and policies and how they might align more closely to the qualitative and contextualised aspirations for research assessment set out in the consensus document. Ismael Ràfols and Jordi Molas-Gallart argue that for Spain this would require significantly changes in the existing research assessment regime towards more university autonomy.

The Agreement on Reforming Research Assessment offers a window of opportunity to transform evaluation systems across Europe. However, in order to implement the qualitative style of evaluation proposed, it will be necessary to modify the processes and sites of evaluation in those countries relying on centralised evaluation systems.

A sea change in European research evaluation

In July 2022 the European Commission (EC) published the Agreement on Reforming Research Assessment, a consensus document coordinated by Science Europe and the European Association of Universities and the end product of a broad consultation with over 350 organizations. This initiative is an EC response to the perception that current research assessment is not only problematic in itself, but has become a barrier to European policies, such as Open Science (OS), Integrity or Responsible Research and Innovation (RRI).

The core commitments of the Agreement focus on the need for evaluation to reflect the plurality of modern research and the varying contexts within which it takes place. It states that evaluation must ‘recognize the diversity of contributions to, and careers in, research’, must be based ‘on qualitative evaluation for which peer review is central’, ‘abandon inappropriate uses (…) of journal and publication-based metrics (…)’ and ‘avoid the use of rankings of research organisations’. In short, the Agreement establishes the primacy of qualitative and peer evaluation to support appropriate contextualisation and diversity of scientific practices and careers.

Last September, it was announced that a Coalition for Advancing Research Assessment will be set up by the end of 2022 with organisations willing to make a commitment to the agreement. More than 125 organizations have already added their signature, including major institutions such as CERN, CNRS, DFG, and the main agencies and university associations in Switzerland, Norway and the Netherlands.

Cultural, institutional and bureaucratic barriers to reform

However, the agreement and its recommendations are by no means a done deal. They require significant cultural change: it will be difficult for some researchers to stop thinking of quality in terms of journal prestige and impact factors. They will also require institutional reforms, such as the formal details that shape criteria in assessments. For example, evaluation forms that now ask for a ‘list of top ten publications’ should now shift to require the ‘most important five scientific contributions.’ However, the most formidable barriers are those processes associated with centralised and bureaucratic governance of evaluation, a critical factor in the example we explore here of Spain.

In Spain universities can only hire and promote lecturers who have been approved by the Spanish Agency for Quality Assessment and Accreditation (ANECA) or equivalent regional agencies. This double-step recruitment process (agency ‘vetting’ before the usual job examination at the university) was implemented in the 2000s to reduce nepotism. The regular research individual incentive for lecturers (‘sexenio’) is also awarded by ANECA. This handling of individual accreditation and assessment by external agencies is rigid and standardised, which means that in practice, evaluation is dependent on journal rankings (derived from impact factors) of the publications submitted. These criteria are in flagrant conflict with international good practices (see DORA or the Leiden Manifesto), as well as with the European agreement since they create strong incentives for narrowing practices towards publication, including gaming.

Devolving autonomy to universities is the only escape from metrics

Is there any possibility of changing these criteria while maintaining a system of individual accreditation and evaluation by external agencies? Can research bureaucracies far removed from the context carry out individual evaluation that ‘recognise the diversity of contributions to, and careers’ in science?

We believe not. Bureaucratic structures require the use of standardised criteria that can be applied quickly and homogeneously to the population assessed. Such a need gives rise to procedures that are incompatible with the appreciation of diversity sought by the Agreement. The pluralisation and flexibility of evaluation criteria cannot be implemented from administrative machinery that must evaluate hundreds of CVs each year, and thus cannot avoid simplifying decisions using metrics. If external agencies tried to adopt contextualized and plural ex-post assessment, they would likely collapse under the time and costs required.

For Spanish universities to be able to adopt the new evaluation criteria, a change in governance is essential. In order to carry out a more plural and flexible evaluation with qualitative criteria, it is necessary to focus on the content of research and this means to evaluate fewer times, at a collective rather than individual level, and from positions closer to the practices assessed.

To achieve this, hiring and promotion decisions should be restored to universities, as is the case in most European countries. Moreover, even on its own terms, the centralised scheme has failed in its goal of preventing ‘academic in-breeding’: after 15 years of application, 73% of Spanish lecturers work at the same university where they read their thesis, and only 2.5% of public university lecturers have a foreign nationality. Yet, is it possible to have autonomy without the risk of favouritism?

Institutional accreditation: university autonomy with governmental checks

Following a proposal of the Andalusian Evaluation and Accreditation agency (DEVA), ‘institutional accreditation’ of university departments or schools is one plausible model of evaluation governance that would allow a change in criteria, while preventing in-breeding. Under the scheme, universities would be given full autonomy in hiring and promotions. On the other hand, every 4 or 6 years, an external evaluation agency would certify that the centres comply with agreed criteria of quality in procedures and results. The system would allow departments to hire and promote candidates with the profiles most suited to their needs. At the same time, the accreditation agency would be able to ascertain that adequate procedures and practices are followed, so that only those centres with demonstrated capacity and results would be allowed to exercise this autonomy.

This proposal does not imply a major legislative change because institutional accreditation is a mechanism that already exists to evaluate university degrees. It would be a matter of adding components for the selection and promotion of professors to the current accreditation schemes.

Reviewing the law on Spanish research assessment

The Spanish parliament is currently discussing a new Law on the University System (so called LOSU). While there is agreement on the need to modernise evaluation, there is not a shared vision about how to do it. In alignment with the European agreement, the current draft law recognises the importance of new university missions and opens options for a more plural evaluation of their contributions. However, in contradiction to this ambition, it maintains the core elements of the existing governance with a focus on individual assessment by external agencies. Although, fortunately, the draft also introduces (but as yet does not deploy) the possibility of institutional accreditation.

Whilst we believe that external accreditation should be simply eliminated, this ambiguity in the draft leaves room for hope. Above all, the new law should aim to prevent the accreditation agencies from attempting to apply the diversifying logic of the European Agreement to bureaucratic criteria. The contradictory logics of the European Agreement and centralised assessment is unlikely to be sustainable and could lead to administrative collapse. On the contrary, reducing the importance of individual accreditation with a simplified procedure, and developing institutional accreditation as the main filter of control, could allow for a much more efficient and contextualised research assessment system that enhances university missions to address the most pressing global challenges.

This blog post is based in an editorial letter just published in the journal El Professional de la Información. 

The content generated on this blog is for information purposes only. This Article gives the views and opinions of the authors and does not reflect the views and opinions of the Impact of Social Science blog (the blog), nor of the London School of Economics and Political Science. Please review our comments policy if you have any concerns on posting a comment below.

Image Credit: Dan Dimmock via Unsplash.