All agree that peer review is an area of scholarly communications that is ripe for innovation. However, it may be that our current conceptualisation of peer review places limits on our progress and ambitions. Jon Treadway highlights four alternative tracks of development, including an increased recognition of the many diverse contributions to the research process, a renewed and widened understanding of what constitutes a “peer”, and a robust evidence base for which techniques and modes of review most consistently benefit authors and the wider scholarly record.
Happy peer review week!
These are exciting times for peer review, which, alongside open data, is perhaps the area of scholarly communications that is most ripe for innovation. The sector is rightly engaged in passionate debate about how peer review should and does work. We have seen and will see more transparency, more pre-publication review, and a wider pool of researchers involved. Products and sites like Publons and Pubpeer should be applauded for the success they have had to date.
Yet academic peer review as we have come to know it – the process by which editors at journals coordinate formal evaluation of articles prior to publication from the researchers most qualified to give it – is not as old as many would imagine. It has been the cornerstone of scientific publishing for years, but is not fundamental, in a sense, to the scientific method.
Our current conceptualisation of peer review may be becoming a limiting factor. Some of the most likely developments in the near term are predicated on outmoded definitions of peers and review, and are unambitious or possibly even detrimental.
Image credit: Light bulb and glowing lights over lily pads in pond by Ky0n Cheng. This work is licensed under a CC0 1.0 license.
For example, the assignment of credit to researchers by adding up the number of reviews they undertake, would not be particularly innovative. It would perpetuate the problems and dissatisfaction with existing metrics. We do not want a situation where impact-hungry institutions use researcher “scores” generated from closed datasets to guide their hiring practices.
Similarly, grading reviewers by the “quality” of journals for which they review poses risks. The standard proxies for quality are gradually losing respect, and the need to be a reviewer for a high-impact journal to go alongside publications in them would be a retrograde step, buttressing the challenges of existing incentive structures.
So what should be done? Many things; with each offering a glimpse of a better, shinier future.
First, we need ways to navigate the full richness of the scholarly record – the different iterations of articles (including preprints, drafts, updated, amended versions of record); the iterations, replications and refutations of results, and also the specific assertions made in interpreting those results; the reviews, comments and annotations made on different platforms.
Second, we need ways of crediting researchers for all of the contributions they make – building on and fully delivering the CRediT proposals, driving beyond them; ensuring that there is appropriate credit for the scientists who curate and maintain datasets; for statisticians; for those who deliver improvements in techniques, and whose fine-grained assertions transform and create whole fields of study but find these contributions currently obscured by incorporation. Credit for peer review, as currently practised, is only a piece of the puzzle, and a relatively inconsequential one at that.
Third, these developments will revolutionise our ability to identify researchers to whom we can turn for insightful comment on and improvements to the outputs of research. Alleviating the pressure on existing researchers and expanding our pool of reviewers is important, but more interesting is whether our definition of “peers” adequately reflects the way researchers make contributions, and accounts for the proliferation of interdisciplinarity, transdisciplinarity and fertile cross-pollination between subject areas. Harry Collins’s concept of “interactional expertise” is an underexplored conceptualisation of how peer review can work in principle, but it is also insightful here on the topic of what constitutes expertise and how we factor it into our processes.
Fourth and finally, we lack a robust evidence base for which techniques and modes of review most consistently benefit authors and the wider scholarly record, and in what context each should be applied. This is an area where more openness from all parties can really provide benefits, and where we can expect advances in machine learning, artificial intelligence, and adaptive learning to offer new and much more nuanced opportunities to incentivise and train editors and reviewers.
Incremental improvements and marginal revolutions should be applauded, but ultimately we must expand our concept of peer review as publication itself changes if we are to realise the greatest innovations.
We do not want to be haunted by the ghouls of our past; being haunted by the spectres of possible futures is much more exciting.
Note: This article gives the views of the author, and not the position of the LSE Impact Blog, nor of the London School of Economics. Please review our comments policy if you have any concerns on posting a comment below.
About the author
Jon Treadway is Chief Operating Officer at Digital Science. Prior to this, he spent five years across the Holtzbrinck Publishing Group companies in a variety of roles and previously worked for the Arts Council and KPMG. Jon is unashamedly Transpontine, a folkie at heart and re-reads Hogfather at least once a year.