Is the peer review process simply a means by which errors are identified and corrected? Or is it a process in which a more constructive dialogue can take place and reviewers and editors may actively contribute to the text? John Rigby, Deborah Cox and Keith Julian have studied the published articles of a social sciences journal and found that the more revisions a paper undergoes, the greater its subsequent recognition in terms of citations.
We recently published an article discussing the effect of how a paper is reviewed and the recognition it receives once published. We provide some statistical evidence from a study of a single journal’s reviewing process over a number of years to show that the more revisions a paper undergoes, the greater its subsequent recognition in terms of citation impact. Our paper suggests that reviewers of a paper and the editor may actively contribute to the text, adding to its interest and leading to greater recognition when it is published.
Our paper is part of a long-running effort to open the “black box” of peer review. Peer review is essential to science but is all too easily seen as a judgemental activity happening at a discrete moment in time. In this very limited conception, peer review is either seen as a decision-making process, where a binary choice is made to accept or reject a paper, or where errors are identified and then removed. But this is very unhelpful. Peer review needs to be recognised not as an event but as a process, often a long one, and one in which a constructive dialogue can take place. It is a process which writers from the business and management field could easily recognise as one of “open innovation” or “co-creation”, although for various reasons one might not go into here, many people would be reluctant to acknowledge it as such.
Our recent research argues that peer review is ubiquitous in the generation of knowledge. We note that the literature on peer review can be divided into two main discussions. On the one hand there are studies that focus on peer review as a decision-making process that needs constantly to be monitored to ensure it fulfils is main objective: the identification of quality and the rejection of poor quality. On the other hand, there is a more critical discussion which is less aimed at optimising a process and more at achieving an understanding of peer review by uncovering the roles played by the different actors and factors that influence the way in which judgements of a paper are made. Our contribution is to this second discussion, and what makes our study the only one of its kind is that we have systematically related features of the pre-publication process – peer review – to the post-publication data on citation. And while a number of contributors to this debate have noted the role of journal peer review in improving the quality, none has demonstrated a statistical link between a paper’s citations and aspects of the peer review process.
Image credit: Wadi Lissa, via Unsplash (licensed under a CC0 1.0 license).
Our initial assumptions were that the amount of time and the number of revisions which a paper undergoes might be related in some way to its later interest to the scientific community. Earlier work of ours suggested that the greater the involvement of other actors in the production of the paper, the less novelty the paper would have, while papers with less involvement would have greater variability of quality; i.e. there would be proportionately more of both highly cited and uncited papers . We also knew from other literature the contrasting claim that the greater the involvement of other actors, the greater the impact of the paper would be. The results of our analysis showed that the amount of time a paper was in review was not related to the subsequent recognition which the paper earned, but we did find that the number of times the paper was revised was a predictor of greater recognition – i.e. the papers were more highly cited.
Our work throws up an interesting paradox – at least an interesting paradox for those who hold that peer review is error correction. If we consider that peer review is simple error removal, the reviewing effort variable in our analysis (the number of times a paper is revised) would either have no statistical relationship with the outcome variable or, as is more likely, it would have a negative relationship in that papers requiring more work at first review (a higher reviewer input) would have less recognition and general interest ultimately as such papers contained more errors to begin with. As we noted above, the analysis of our data shows a positive connection between the effort of reviewing and the outcome variable. Under the assumption that peer review is merely error checking, this association would be a paradox which one could state thus: manuscripts at review that were thought to be in more need of correction were, at publication found to be of greater interest. We have therefore concluded – albeit on the basis of our very small study – that peer review is more than mere error correction.
We believe there should be more research on this topic and are planning to do some more work shortly. Peer review is a sensitive subject, though, and if anything it is closer to the study of political processes than we might like to admit.
This blog post is based on the authors’ article, “Journal peer review: a bar or bridge? An analysis of a paper’s revision history and turnaround time, and the effect on citation”, published in Scientometrics (DOI: 10.1007/s11192-017-2630-5).
Note: This article gives the views of the authors, and not the position of the LSE Impact Blog, nor of the London School of Economics. Please review our comments policy if you have any concerns on posting a comment below.
About the authors
John Rigby read history at Cambridge and then worked in IT before completing his PhD at the University of Manchester, studying under Professor Luke Georghiou and Dr Mark Boden of PREST, and Dr Sue Tindal of the Building Research Establishment, and writing his thesis on the rationale, rhetoric and impact of an information and advice dissemination programme for energy efficiency. John works in the Manchester Institute of Innovation Research, the University’s main research centre in science and technology policy. John is one of the Alliance Manchester Business School’s postgraduate research coordinators. His interest, which he has shared with Keith Julian, is in peer review processes, and they have developed their understanding through over a decade of work on the subject.
Deborah Cox studied social sciences at degree level and has a master’s degree in Information Management from Sheffield University. She is also a qualified teacher and has a teaching degree in Adult, Further and Higher Education from Manchester Metropolitan University. She has been working in the field of STI policy research for over 20 years at Manchester and has experience of leading and managing successful transnational research projects. Deborah has co-authored a range of publications in the field of STI policy including books, journal articles, OECD briefings and research reports.
Keith Julian read chemistry at Oxford where he stayed to take his DPhil in 1962 after which he worked in industry ICI, Foseco Minsep, Chloride Group, International Paint (Courtaulds Group) and BICC, retiring from BICC Cables, part of BICC plc, as Director of Engineering in 1996, after which he ran his own company, KJMetrics, as a consulting statistician and process engineer until 2006, and collaborated with a number of UK universities including Newcastle and Manchester at which of he held visiting professor positions. It was at Manchester that Keith met John and their exploration of peer review processes developed from there through a series of collaborations.
So badly written papers should be regularly updated and improved so as to get the maximum of attention. Logical but weird.