The Chinese government recently announced that research assessment in China should no longer be predominantly focused on metrics, Web of Science based indicators and what has become known as ‘SCI worship’. In this post Lin Zhang and Gunnar Sivertsen discuss how China’s new research policy might be implemented and the parallels it has to recent attempts to reform other national and international research systems.
A radical reform of research evaluation and funding in China was recently launched in two prescriptive policy documents published by the Ministry of Science and Technology and the Ministry of Education. China now moves from a strong focus on Web of Science-based indicators towards a more balanced combination of qualitative and quantitative research evaluation, with one third of all publications to be oriented towards domestic journals. Universities are urged to implement the policy locally by the end of July at the latest. How to do it, and the possible consequences, have aroused intense discussion among Chinese academics and gained worldwide attention and debate.
This change has not come out of the blue. In 2016, President Xi Jinping called for reform towards a more comprehensive evaluation system for individual researchers. Further, in 2018, a document issued by three ministries and two national central institutions specifically proposed moving away from the “Four only” phenomenon of recognising and rewarding “only papers, only titles, only diplomas and only awards”.
China now moves from a strong focus on Web of Science-based indicators towards a more balanced combination of qualitative and quantitative research evaluation
The reform explicitly turns away from what has been called “SCI worship” (Science Citation Index being the original name of Web of Science, WoS). At all levels in the Chinese research system there was a strong focus on Journal Impact Factors, JCR Quartiles and ESI Highly Cited Papers. The uniform ‘currency’ of WoS-based indicators has been used in research evaluation, staff employment, career promotion, awards, university or disciplinary rankings, funding and resource allocation. Even individual cash incentives for WoS publications are widespread.
The effects have been twofold. On the one hand, Chinese researchers have been encouraged to productively engage with the global research community. It was partly this focus that helped China surpass the USA as the largest contributor to international scientific journals. On the other hand, the heavy reliance on “SCI papers” has been viewed as a form of goal displacement, leading individual researchers and institutions to pursue high numbers of publications at the expense of their quality, societal value and at times research integrity.
The need for new research assessment infrastructures
The new policy aims to restore “the scientific spirit, innovation quality, and societal contribution” of research and to “promote the return of universities to their original academic aims”. As we understand the policy documents outline three main measures to reach these aims:
- Farewell to “SCI worship”. Indicators based on WoS will not be applied directly in evaluation and funding at any level. An alternative citation index with Chinese characteristics and international influence will be established.
- From metrics to peer review. A new focus on novelty, scientific value, research integrity, innovation potential and societal outcomes will replace the “paper only” orientation in panel evaluations. Publications will be presented for review as a limited set of “representative work” with explicit relevance for the evaluation. Number of publications and journal impact factors will not count any more.
- Local relevance. Publications in high-quality Chinese journals will be encouraged, and the development of such journals will be supported.
The reform has similarities with initiatives in other parts of the world, such as the DORA declaration on research assessment, the Leiden Manifesto for research metrics, and the EU policy for Responsible Research and Innovation . None of these initiatives have been easily implemented or provided immediate solutions to the problems they were designed to resolve. In our view, the need for national coordination and services to replace the WoS presents similar challenges in all three of the main areas targeted by the reform.
Farewell to “SCI worship”
By moving away from a commercial product-based WoS (or Scopus) approach for research evaluation and funding system, China is empowering its own academic communities, research institutions and funding organisations to develop a self-determined and self-organised criteria-based system. However, to fulfil this transition, an integrated research information system and a national journal evaluation system is still needed.
In effect, the policy demands a broader perspective of research assessment on novelty, scientific value, research integrity, innovation potential and societal outcomes. To be informed about and be able to compare such achievements requires comprehensive sources of information to supplement WoS. These sources could be integrated in a national Current Research Information System (CRIS), following the examples of; Brazil, Czech Republic, Finland, New Zealand, and Norway.
To replace WoS as a standard, there is also a need to create a comprehensive and dynamic list of recognised journals. The list must be dynamic to reflect a changing journal market, and it needs to be organised to represent a balanced influence of expert advice by all disciplines in China through inter-institutional representative bodies. The same organisation of expert advice is needed if extra weight is to be given to specific selections of journals on the comprehensive list. Examples of such dynamic lists already exist in several non-English speaking countries in Africa, Asia, Europe and Latin-America, e.g. the Latindex and the Nordic list.
From metrics to peer review
The new policy targets evaluation and funding at all levels in the Chinese research and innovation system. This would suggest that methods for individual-level research assessment will also be used as a model for evaluation at aggregate levels (institutions, thematic programs, research sectors). As an example, the new policy requires a small maximum number of representative publications to be read and evaluated at all levels. The new policy needs to differentiate more clearly between appropriate methods at different levels of aggregation. Depending on the purpose, metrics can be useful at aggregate levels, and the data and indicators need not be limited to WoS. When moving from metrics to peer review, a multi-level application model for roles and procedures in research evaluation will be needed.
Peer review allows for formative, not only summative, evaluations. A formative evaluation learns from the past (strengths and weaknesses), looks forward (opportunities, threats) and serves strategic development. A summative evaluation looks at past performance, checks whether goals or expectations have been reached, and serves accountancy, decisions and/or resource allocation. Summative evaluation has dominated in China. The idea of formative evaluation is not present in the new policy. National coordination could allow for organisational learning at the aggregate level of institutions and for inter-institutional disciplinary collaboration.
Local relevance
Chinese researchers are now encouraged to publish more in domestic journals: “In principle, when researchers provide representative publication lists, papers from domestic journals should account for at least one third of all the publications”. Several provincial governments and universities in China have already incorporated this new principle into their policies. In our view, the principle needs to be applied with differentiation according to field, type and purpose of research.
There has been a concern for some years that research expected to be useful for Chinese society is published in English in very distant journals. There has been a lively debate about over the fact that one of the first scientific articles carrying an early warning of the Coronavirus was published by Chinese scientists in a Western international journal before the general public in China was informed about the epidemic. This reaction in the general public is understandable, but experts in the field will know that international and local publishing cannot replace each other. Time has also shown that global exchange of information and advice is crucial to stop the Corona epidemic itself as it reaches other countries and continents.
Most scientific publications are still published in Chinese in China, with variations among fields. Some fields are internationally visible and impactful, others are not. Some researchers are happy to leave behind the policy of globalisation. Others are concerned that support for collaborating and publishing abroad will be taken away from them. In setting out these reforms, China will ultimately have to negotiate these tensions and develop a differentiated and dynamic balance between local relevance and globalisation, peer review and metrics based research assessment.
This blog post is based on the authors’ article, “The New Research Assessment Reform in China and Its Implementation”, published in Scholarly Assessment Reports.
Note: This article gives the views of the authors, and not the position of the LSE Impact Blog, nor of the London School of Economics. Please review our comments policy if you have any concerns on posting a comment below
Image Credit: Howchou, Wuhan University Sakrua Castle 2, via Wikimedia Commons (CC BY 4.0)
