Open peer review is moving into the mainstream, but it is often poorly understood and surveys of researcher attitudes show important barriers to implementation. Tony Ross-Hellauer provides an overview of work conducted as part of an OpenAIRE2020 project to offer clarity on OPR, and issues an open call to publishers and researchers interested in OPR to come together to share data and scientifically explore the efficacy of OPR systems as part of an Open Peer Review Assessment Framework.
Informally, peer review is just the development of nascent ideas and theories through critical discussion with others. As such it is as old as knowledge creation itself. Its stricter sense – the formal scholarly publishing process where an editor sends copies of a manuscript to people judged knowledgeable enough to be able to comment on its suitability for publication – is more recent. In this mode it has been the default of academic publishing only since the mid-20th century.
Peer review serves to validate the soundness, substance and originality of a work, or to help improve it until it meets required standards for these criteria. Peer review, in its current form, is typically:
- Anonymous – either the author doesn’t know who the reviewer is (single-blind) or author and reviewer are unknown to each other (double-blind).
- Hidden – the process takes place behind closed doors (or, rather, password privileges) and reviews are not published.
- Selective – reviewers are chosen by the editor.
Studies have shown that although academics are, on the whole, fairly content with peer review, they think it could work better. Here are some of its problems:
- Time – peer review often takes a long, long time. Could open review help speed up the process?
- Accountability – the anonymity of reviewing, although in principle meritocratic (as junior researchers can criticise the work of luminaries without fear of reprisals), also makes it unaccountable (some say Kafkaesque). Should professionals cast judgements in secret? Shouldn’t they be prepared to stand openly by what they believe? Relatedly, would reviews conducted in public be more constructive and less confrontational?
- Bias – given the specialised nature of academia, a researcher’s nearest “peers” will often be known to them as either friends or rivals. This fact naturally leads to concern that rejection or acceptance might sometimes have social, rather than scientific, grounds. Even where authors’ names are withheld, it is often clear from the research itself (or a cursory Google) who the author is. If reviews were public, such biases might be further suppressed.
- Scams – that the peer process takes place behind closed doors also likely aids peer review scams in avoiding detection.
- Incentive – reviewing, done well, is hard work. If reviews were open, busy academics and researchers could take credit for them, demonstrating experience, community involvement, and impact.
- Wasted effort – reviewer comments often add context or point to areas for future work. Reviewer disagreements can expose areas of tension in a theory or argument. Readers may find such information helpful and yet, at present, this potentially valuable additional information is wasted. Furthermore, as rejected papers are usually resubmitted elsewhere (to be reviewed all over again), there is a duplication of effort – the same paper might be reviewed multiple times.
As the open science agenda has taken hold, “open peer review” has been proposed as a solution to some of these problems. By bringing peer review into line with the aims of open science, proponents aim to bring greater transparency, flexibility, inclusivity and/or accountability to the process. Various innovative publishers, including F1000, BioMed Central, PeerJ, and Copernicus Publications, already implement systems that identify themselves as OPR. However, such systems differ widely across publishers, and indeed – as has been consistently noted – OPR has neither a standardised definition nor an agreed schema of its features and implementations.
While the term is used by some to refer to peer review where the identities of both author and reviewer are disclosed to each other, for others it signifies systems where reviewer reports are published alongside articles. For others it signifies both of these conditions, and for yet others it describes systems where not only “invited experts” are able to comment. For still others, it includes a variety of combinations of these and other novel methods. These differing flavours of OPR include independent factors (open identities, open reports, open participation, etc.) which usually have no necessary connection to each other, and very different benefits and drawbacks. Evaluation of the efficacy of these differing variables and hence comparison between differing systems can be problematic as it is often unclear which distinct configuration of OPR is under discussion.
Defining open peer review
To bring clarity to how the term is used, I analysed 122 separate definitions from the literature, identifying seven different key traits of OPR:
- Open identities – authors and reviewers are aware of each other’s identity.
- Open reports – review reports are published alongside the relevant article.
- Open participation – the wider community are able to contribute to the review process.
- Open interaction – direct reciprocal discussion between author(s) and reviewers, and/or between reviewers, is allowed and encouraged.
- Open pre-review manuscripts – manuscripts are made immediately available (e.g. via preprint servers like arXiv) in advance of formal peer review procedures.
- Open final-version commenting – review or commenting on final “version of record” publications.
- Open platforms (“decoupled review”) – review is facilitated by a different organisational entity than the venue of publication.
A total of 22 unique configurations of these traits were found amongst the various definitions (i.e. 22 distinct definitions of OPR in the reviewed literature). Across all definitions, the core elements are open identities and open reports, with one or both elements present in over 95% of the definitions examined. Among the other elements, open participation is the next most common element, and should perhaps be considered a core trait in the social sciences and humanities (where it appears in more than 50% of definitions). Further secondary elements are open interaction and open pre-review manuscripts. Fringe elements include open final version commenting and open platforms.
Figure 1: Distribution of OPR traits amongst definitions. Source: Ross-Hellauer, T. (2017) “What is open peer review? A systematic review [version 2; referees: 1 approved, 3 approved with reservations]“. F1000Research. This work is licensed under a CC BY 4.0 license.
Figure 2: Unique configurations of OPR traits within definitions. Source: Ross-Hellauer, T. (2017) “What is open peer review? A systematic review [version 2; referees: 1 approved, 3 approved with reservations]“. F1000Research. This work is licensed under a CC BY 4.0 license.
Given that OPR is such a contested concept, in our view the only sensible way forward is to acknowledge the ambiguity of this term. Hence, I propose to define OPR as an umbrella term for a number of overlapping ways that peer review models can be adapted in line with the aims of open science. Quantifying the ambiguity of usage and mapping the distinct traits enables future discussion to start from a firmer basis that: (1) acknowledges that people often mean different things when they use this term; and (2) clarifies in advance exactly which OPR traits are under discussion.
Being clear about these distinct traits will enable us to treat the ambiguity of OPR as a feature and not a bug. The large number of possible configurations of options presents a toolkit for differing communities to construct open peer review systems that reflect their own needs, preferences, and goals. The finding that there seems to be a difference in interpretations between disciplines (for example, that open participation seems more central to conceptions of OPR in SSH than science, technology, engineering and maths) reinforces this view. Moreover, disambiguating these traits will enable more focused analysis of the extent to which these traits are actually effective in countering the problems it is claimed they address. This is particularly urgent because, as the analysis showed, there is often little evidence to support or refute many of these claims.
What do authors, reviewers and editors think of OPR?
A second, related problem is a lack of research detailing stakeholders’ attitudes towards these different systems. It is a truism that technological change is easy in comparison to cultural change, and this certainly holds for OPR. The diverse innovations represented by the term will remain niche possibilities unless accepted and adopted by the mainstream. Questions hence arise: what are scholars’ current attitudes to peer review? Where do they believe it can be improved? What are their opinions on the various aspects of OPR? What are the cultural barriers that stand in the way of the uptake of such systems? What are the current levels of experience amongst stakeholders with OPR? To date no systematic analysis of these questions has been undertaken.
To address this knowledge gap, the open science initiative OpenAIRE conducted an online survey of 3062 editors, authors, and reviewers during September and October 2016. The full survey results (currently under review for publication but already available as preprint) show the majority of respondents to be in favour of OPR becoming mainstream scholarly practice, as is the case for other open science practices, like open access and open data. We also observed surprisingly high levels of experience with OPR, with three out of four (76.2%) respondents reporting having taken part in an OPR process as author, reviewer, or editor. There were also high levels of support for most traits of OPR, particularly open interaction, open reports, and final-version commenting. Respondents were against opening reviewer identities to authors, however, with more than half believing it would make peer review worse. Overall satisfaction with the peer review system used by scholarly journals seems to significantly vary across disciplines.
Figure 3: Will XXXX make peer review better, worse, or have no effect? Source: Ross-Hellauer, T., Deppe, A., Schmidt, B. (2017) “OpenAIRE survey on open peer review: Attitudes and experience amongst editors, authors and reviewers“. Zenodo. This work is licensed under a CC BY 4.0 license.
Taken together, these findings are very encouraging for OPR’s prospects of moving mainstream but indicate that due care must be taken to avoid a “one-size-fits-all” solution and to tailor such systems to differing (especially disciplinary) contexts. More research is also needed. OPR is an evolving phenomenon and hence future studies are to be encouraged, especially to further explore differences between disciplines and monitor the evolution of attitudes.
OPR Assessment Framework
OPR is a diverse cluster of interrelated yet distinct innovations that aim to bring open science principles like transparency, accountability, and inclusivity to the peer review process. It is moving mainstream. But there remain significant hurdles to widespread acceptance.
Variety and experimentation are key to the success of OPR. The differing kinds of OPR have a lot of potential to improve peer review, but it is clear that no one model will fit all disciplines or overcomes all obstacles. Amongst variants it is important we are able to distinguish readily between them, and to recognise what actually works in which contexts.
Answering such questions in a systematic way should be the next goal for those interested in OPR. We need evidence to show what works in which circumstances, and this is sadly lacking at present. I hence propose the development of an OPR Assessment Framework, which would seek to systematically evaluate the efficacy of differing OPR systems across publishers and OPR models. This would be greatly helped by making available open datasets of open peer reviews and associated metadata. The PEERE network’s recent achievements in data sharing are important, but OPR has not yet been a core concern of its work and, unfortunately, it seems that access to the shared data is restricted to consortium members only at present. Ideally we would open all suitably anonymised information about peer review, via open datasets according to shared standards, to the wider community of scholars to maximise research.
Following agreement on a common understanding of OPR, key priorities for research should then be established and a framework for their analysis agreed. Such “open questions” for OPR include:
- In the case of open identities, it seems stakeholders are reflecting common fears that either reviewers will hold back valid criticisms for fear of offending (especially senior) peers, or that forthright reviewers will be subject to future reprisals. Are such fears, often based on anecdata, valid? Do researchers act in such ways in OPR systems? If so, how could this be avoided?
- Are open identities reviewers less likely to criticise? Does this make for better or worse reviews?
- Are reviewers less likely to review under open identities? Do they take longer to write and/or submit their reports? How does this impact costs?
- How are underrepresented groups (based on geographical region, primary language, gender, less-prestigious institutions) impacted by open identities?
- To what extent are reviews actually improved or made worse by revealing reviewer names?
- Are any of these factors mitigated or changed by delaying revealing reviewer names (for instance until after acceptance)?
- Are reviewers less likely to review under open reports? Do they take longer to write and/or submit their reports? How does this impact costs?
- How does open reports affect the quality, substance, and length of review reports?
- Does access to published reports actually aid junior researchers beginning to peer review (as has been theorised and reported anecdotally)? To what extent?
- To what extent does publishing reports with reviewer identities raise recognition and reward of the work of peer reviewers?
- To what extent are published referee reports cited or otherwise reused?
- Do open-participation peer review systems attract more or fewer reviewers from beyond disciplinary “siloes”?
- Are conflicts of interest more likely in open participation systems?
- How might low-levels of uptake for open participation and final-version commenting be overcome?
- What levels of editor or similar intermediary will continue to be required to find, engage, and motivate reviewers?
- How might reviewers be motivated to engage in open participation processes without such mediation?
- Do interactive peer review systems such as those at EMBO, eLife and Frontiers lead to improved review processes (e.g. higher review quality, better error detection, etc.)?
- How are costs and review times impacted?
- Are the findings of levels of experience with and attitudes towards OPR reported in the survey results above consistent across studies?
- Which specific OPR systems (run via journals or third-party services) do users (within differing disciplines) most prefer?
- What measures might further incentivise uptake of OPR?
- How fixed are attitudes to the various facets of OPR and how might they be changed?
- What are attitudes to OPR for research outputs other than journal articles (e.g. data, software, conference submissions, project proposals, etc.)?
- How have attitudes changed over time? As OPR gains familiarity amongst researchers and is further adopted in scholarly publishing, do attitudes towards specific elements like open identities change? In what ways?
- To what extent are attitudes and practices regarding OPR consistent? What factors influence any discrepancies?
This blog post is based on the author’s articles, “What is open peer review? A systematic review [version 2; referees: 1 approved, 3 approved with reservations]“, currently available at F1000Research (DOI: 10.12688/f1000research.11369.2); and “OpenAIRE survey on open peer review: Attitudes and experience amongst editors, authors and reviewers“, co-authored with Arvid Deppe and Birgit Schmidt and currently available at Zenodo (DOI: 10.5281/zenodo.570864).
Featured image credit: open neon by Bill Smith. This work is licensed under a CC BY 2.0 license.
Note: This article gives the views of the authors, and not the position of the LSE Impact Blog, nor of the London School of Economics. Please review our comments policy if you have any concerns on posting a comment below.
About the author
Tony Ross-Hellauer will shortly join the Social Computing Research Group at Know-Center as a senior postdoctoral researcher. His main research interests are open science models and infrastructures, science policy, alternative models for peer review, and philosophy of technology. He has a PhD in Information Studies from the University of Glasgow, in addition to an MA (Philosophy) and MSc (Information and Library Studies). Tony has worked in a number of EU-funded projects and is actively involved in open science advocacy and community-building. As Scientific Manager for OpenAIRE at the University of Goettingen, Tony was responsible for OpenAIRE’s outreach strategy, scientific direction and coordination of its pan-European network of National Open Access Desks (NOADs). Tony has published widely on open science, open access, peer review and library science in both peer-reviewed and popular science outlets, and regularly acts as PC member, co-organiser or co-chair of a number of workshops and conferences on topics related to open science.
Good article. The point about the peer review process being done behind closed doors is a valid one and reminds me of PhD funding panels. The applicant doesn’t know who are on these panels and how or why they make decisions on allocating funding. Especially when applicants with lower grades are given funding over those with higher ones. The word ‘scam’ does indeed spring to mind….