This week is Peer Review Week 2018, a global event celebrating the essential role that peer review plays in maintaining scientific quality. Peer Review Week “brings together individuals, institutions, and organisations committed to sharing the central message that good peer review, whatever shape or form it might take, is critical to scholarly communications”.
In addition to the new posts to have published this week, the reading list below brings together a selection of the fantastic contributions on the subject of peer review to have featured on the Impact Blog over the past couple of years.
You can keep track of all of the fascinating commentary, research, discussion, and debate, including on the 2018 theme of “Diversity in Peer Review”, by following the hashtag #PeerReviewWeek18. Enjoy!
Unhelpful, caustic and slow: the academic community should rethink the way publications are reviewed
The current review system for many academic articles is flawed, hindering the publication of excellent, timely research. Thomas Wagenknecht offers some solutions to the current system, including encouraging associate editors to use their authority to mitigate the impact of bad reviewers, shortening the entire process, and increasing peer reviewer education during PhD programmes. However, there are also opportunities for more significant reforms, by adopting post-publication peer review and by exploiting new distributed ledger technologies.
There is little evidence to suggest peer reviewer training programmes improve the quality of reviews
A number of new peer reviewer training programmes have launched, promising to help early-career researchers learn how to do peer review, review more efficiently, and connect with editors at top journals. This follows an expressed need from graduate students and postdocs for precisely this sort of training. But can these new programmes deliver? And as many providers suggest moves towards a subscription-based model, are they worth paying for? Shaun Khoo examines the evidence and finds little to suggest that peer reviewer training programmes actually improve the quality of article reviews.
Is peer review bad for your mental health?
Amidst fears of a mental health crisis in higher education, to what extent is the peer review process a contributing factor? It’s a process fraught with uncertainty, as authors try to forge something constructive from mixed feedback or downright unhelpful comments. Helen Kara stresses the importance of being aware of the effects of uncertainty and taking steps to reduce its impact. Focus on what you can control, prepare for different outcomes, acknowledge how you’re feeling, and make sure to practise self-care.
The more revisions a paper undergoes, the greater its subsequent recognition in terms of citations
Is peer review simply a means by which errors are identified and corrected? Or is it a process in which a more constructive dialogue can take place and reviewers and editors may actively contribute to the text? John Rigby, Deborah Cox and Keith Julian have studied the published articles of a social sciences journal and found that the more revisions a paper undergoes, the greater its subsequent recognition in terms of citations.
What are the barriers to post-publication peer review?
Post-publication peer review emerged in response to increased calls for continuous moderation of the published research literature, consistent questioning of the functionality of the traditional peer review model, and a recognition that scientific discourse does not stop at the point of publication. However, uptake remains low overall. Jon Tennant sets out what the barriers to more widespread adoption of post-publication peer review have been and proposes potential solutions for each.
Writing a peer review is a structured process that can be learned and improved – 12 steps to follow
Peer review not only helps to maintain the quality and integrity of scientific literature but is also key to a researcher’s development. As well as offering opportunities to keep abreast of current research and hone critical analysis skills, writing a peer review can teach you how to spot common flaws in research papers and improve your own chances of being a successful published author. Jo Wilkinson asked an expert panel of researchers what steps they take to ensure a rigorous and robust review. Their advice has been compiled into the following 12 steps, relevant to both first-time peer reviewers and those keen to brush up on their skills.
We have the technology to save peer review – now it is up to our communities to implement it
Jon Tennant, Daniel Graziotin and Sarah Kearns consider what can be done to address the various shortcomings and problems of the peer review process. While there is obviously substantial scope for improvement, none of the ideas proposed here are beyond our current technical and social means. The key challenge may lie in galvanising our scholarly communities.
Open peer review: bringing transparency, accountability, and inclusivity to the peer review process
Open peer review is moving into the mainstream, but it is often poorly understood and surveys of researcher attitudes show important barriers to implementation. Tony Ross-Hellauer provides an overview of work conducted as part of an OpenAIRE2020 project to offer clarity on OPR, and issues an open call to publishers and researchers interested in OPR to come together to share data and scientifically explore the efficacy of OPR systems as part of an Open Peer Review Assessment Framework.
Our current conceptualisation of peer review must be expanded if we’re to realise the greatest innovations
All agree that peer review is an area of scholarly communications that is ripe for innovation. However, it may be that our current conceptualisation of peer review places limits on our progress and ambitions. Jon Treadway highlights four alternative tracks of development, including an increased recognition of the many diverse contributions to the research process, a renewed and widened understanding of what constitutes a “peer”, and a robust evidence base for which techniques and modes of review most consistently benefit authors and the wider scholarly record.
What are researchers’ expectations and experiences of the peer review process? Findings from recent research
What do researchers expect of the peer review process? And do their experiences deliver on these expectations? Elaine Devine reports on the findings of recent research that sought answers to these questions. Researchers feel strongly that peer review should, and mostly does, improve the quality of research articles; that incidences of academic fraud are not detected as much as expected; and that a relatively high prevalence of regional and seniority bias exists in peer review.
Amidst criticism of the peer review process, the valuable contributions of reviewers should be defended
As flaws in the peer review process are highlighted and calls for reform become more frequent, it may be tempting for some to denigrate and dismiss the contributions of the reviewers themselves. Maxine David has been witness to this and here makes an appeal to give space to recognise those who offer their time and expertise voluntarily and generously.
Peer review processes risk stifling creativity and limiting opportunities for game-changing scientific discoveries
According to Don Braben and Rod Dowler, the current peer review process actually serves as a blocker to more radical research, stifling creativity and limiting opportunities for game-changing discoveries. Obviously peer review should not be abandoned entirely, but it is time to recognise the need for a separate category of highly innovative research with appropriate funding.
The next stage of SocArXiv’s development: bringing greater transparency and efficiency to the peer review process
Up to now SocArXiv has operated alongside the peer-review journal system rather than seriously disrupting it. Looking ahead to the next stage of its development, Philip Cohen considers how SocArXiv might challenge the peer review system to be more efficient and transparent, firstly by confronting the bias that leads many who benefit from the status quo to characterise mooted alternatives as extreme. The value and implications of openness in the system must be debated, as should potentially more disruptive innovations such as non-exclusive review and publication or crowdsourcing reviews.
How to survive the cruel world of peer-reviewed funding applications
With government funding and industry support for research either static or falling, the grant funding environment has become increasingly competitive. Most funding goes to those in secure employment who have been in academia for some time, making the outlook particularly grim for early-career researchers. Jonathan O’Donnell sets out some practical advice for early-career researchers competing for grant funding; starting with what can be learnt by serving as a reviewer of applications, how collaborating with others can help, and finally considering what the options are outside of the peer review system. There is also advice for the funding agencies themselves on what they can do to make the process more dynamic and efficient.
Formal recognition for peer review will propel research forward
A number of high-profile recent cases have led to increased public and media scepticism over the quality and integrity of research. Peer review remains the gold standard for ensuring that quality and integrity but is reliant on voluntary, unrewarded contributions. Andrew Preston and Tom Culley argue that formally recognising and rewarding peer review efforts would be a great first step towards restoring balance to a research ecosystem that offers experts disproportionate rewards for publishing while offering no formal incentives to the same experts relied upon to filter out the false, fraudulent or misleading.
Manipulating the peer review process: why it happens and how it might be prevented
Peer review is upheld as the best way to evaluate academic research ahead of publication. Yet the peer review process has been consistently targeted and manipulated by authors, reviewers and even editors. Sneha Kulkarni reveals how this happens and what might be done to prevent it, considering the merits of different peer review models but also the target-driven culture of academia that leads some to engage in misconduct.
Why are interdisciplinary research proposals less likely to be funded? Lack of adequate peer review may be a factor
Interdisciplinary research is less likely to be funded than discipline-based research proposals. Gabriele Bammer looks at how interdisciplinary research is currently peer reviewed and argues different review processes may be required to do justice to the different kinds of interdisciplinarity. Discipline-based researchers may be ill-equipped to evaluate the integrative processes that an interdisciplinary proposal plans to use.
What do mathematicians think about their journals? Peer review quality tops list of stated issues
Cameron Neylon, David Michael Roberts and Mark C Wilson have conducted a large-scale survey of what mathematicians think of their scholarly publishing options and what improvements are required. Covering topics like open access, peer review and editorial processes, the survey reveals some fascinating insights into the current scholarly communication system and what changes could be made to make it better.
Breaking the traditional mould of peer review: Why we need a more transparent process of research evaluation
Jon Tennant takes a look at the transformations underway aimed at tackling the widespread dissatisfaction with the system of peer review. He provides further background on the platform ScienceOpen, which seeks to enable a process of transparent, self-regulating, peer review, where knowledge sharing is encouraged, valued, and rewarded. By adopting a more transparent process of research evaluation, we move one step closer towards a fairer and democratic research process.
How to write a peer review to improve scholarship: Do unto others as you would wish them do unto you
Peer review functions as a quality-assurance mechanism which also aims to improve the scientific process as a whole. But few reviewers are provided with any training or mentoring on how to undertake a review. Hugh McLaughlin offers clarification on the process and the objective of peer review feedback and stresses the need for specific critical analysis for authors, which should be done in a supportive manner.
The nuts and bolts of peer review: what are the benefits for an early-career researcher?
For many early career researchers, the trepidation in submitting a first review is hard to overcome. Jillian Hart shares her thoughts following a workshop run by Sense About Science aimed at uncovering the peer review process and tackling those anxieties. She reflects on the benefits for researchers, collectively and individually, in being part of a community of peer reviewers. In this age of consumerism and market-driven strategy, it is ultimately positive that such a system exists where people use their own time, unpaid, to critically review others’ work.
