Open peer review is an unfamiliar experience for many academics, with the added transparency acting as something of a shock to the system. Cristina Costa argues that the change could facilitate a welcome shift away from ‘peer view as monologue’ towards a more dialogical approach.
A couple of months ago David Walker asked me to review a paper for a new Academic Journal of which he is one of the editors. He told me it was an open access journal focused on practice and I immediately said yes! I just cannot say no to open access or perspectives on practice. I just can’t! So I became a reviewer of the Journal of Perspectives in Applied Academic Practice.
As I started going through the first paper I had to review, I noticed that the author’s name was disclosed. There was even a short biography about his academic career. I was intrigued, almost shocked, I must confess. I immediately emailed David reporting the “tragedy” of having learnt the name and background of the author whose article I was about to review. David’s answer was something like this:
You didn’t read the guidelines, did you?!
Cough… well, actually I did, but I went directly to “conducting the review” section, ignoring the opening paragraph of the Reviewers Guidelines. How scholastic of me. So here it is:
Journal of Perspectives in Applied Academic Practice (JPAAP) journal uses open peer review process, meaning that the identities of the authors and those of the reviewers will be made known to each other during the review process in the following way: the reviewers will be fully aware of the name, position and institution of the authors of the manuscript they currently review, and the authors will be given a signed review with the name, position and institution of the reviewer.
I got the shock of my life at first, but then I decided to give it a go. This was the first time I’d been involved in open peer reviewing and, as I think it’s important to put into practice what we preach, I went for it. Here are some reflective points about the process of conducting an open peer review. I aimed to:
- be on my best behaviour – I made sure I allocated plenty of time to read and digest the paper. I read it several times. I tried to understand and deconstruct it the best way I could, before I submitted the review
- give thorough feedback – I tried to justify every point I made (I think I achieved that better in the second paper I reviewed)
- provide constructive and also friendly feedback – Nothing annoys me more than reviews that are dry and harsh in their comments. I tried to use language that aimed to provide suggestions and stimulate new thinking. I think I still have some work to do in this area, but I hope I’m getting there
- read the paper as it is written, not as I would write it – I think that’s a crucial point in any type of review, but one that is often forgotten. I have felt many times that reviews were made on the assumption that the article should be written in the style and perspective of the reviewer rather than that of the author’s
You may think that this doesn’t differ at all from any review process, and in fact it should not. But the feelings and the thoughts that go through your mind as you disclose your identity to the author are both of vulnerability and commitment to do a good job. [Not much different from the feelings of the author of the article who submits his/her work to you, hey?! So we are on the same boat!]
There is something about the open peer review process. With transparency comes visibility, a more acute sense of responsibility of your role as reviewer and maybe the fact that your reputation is on the line matters too! And that can only be good.
However, I still have unanswered questions that in a sense do show my vulnerability as a reviewer.
- would I feel the same if I knew/have worked with the author whose paper I was reviewing?
- Would I feel comfortable giving them my feedback?
- Would I be influenced or even intimidated by my knowledge of their practice/research
- Would I be too tough, or too soft, on them?
I guess you can always refuse to review someone’s paper if those feelings arise and you are not comfortable with it… but these questions did come to mind.
As we move towards more open peer reviewing processes, and I hope more initiatives like this start to emerge, I’d like to see more dialogue between the reviewer and the author. So far it’s still a monologue, and since we disclosed identities, could we also open up the discussion? ~ just a thought.
This was originally posted on Cristina’s personal blog and is reposted with permission.
Note: This article gives the views of the author, and not the position of the Impact of Social Science blog, nor of the London School of Economics. Please review our Comments Policy if you have any concerns on posting a comment below.
Cristina Costa is a Lecturer in Lifelong Learning (TEL) at the University of Strathclyde, Scotland.
Thank you Cristina for sharing your positive experience on such a difference peer review model. Personally, I have been advocating open peer review for many years and have been receiving enthusiastic feedback by most of my colleagues from the areas of physics and biology, but also from less applied sciences. This motivated me to collaborate with other similar-minded scholars to develop an open peer review platform that is author-guided and journal-independent. This means that, more than opening the peer review process in the way you describe, we empower authors to choose their reviewers themselves and to even do this before they submit their manuscript to a specific academic journal. Interested readers can learn more about this particular review process here, watch our brief introductory video, and read a recent article of mine for the benefits of changing the scientific evaluation process, posted here on the LSE blog.
From this perspective, I would like to share my thoughts on your “unanswered questions”.
First, I believe that open peer review should be accompanied by a second iterative step where the community openly rates and comments on the reviews themselves. This is very important to ensure that favorable or biased reviews will be noticed and potentially “punished” by expert readers. This will make reviewers think twice before making unsubstantiated comments or favour certain work due to a personal relationship with the author(s). It certainly makes accepting to review a paper more hard as you are no longer protected by anonymity, but, as you have experienced, improves the quality of the reviews, fosters collaboration between authors and reviewers for the benefit of science, and avoids unethical conduct permitted by the current system. On the other hand, our proposed system may also make reviews easier as specific scholars can be asked to comment on a concrete part of the paper within their area of expertise, instead of deciding on the fate of the entire work. Practically, it allows authors to collect an infinite number of brief reports from specific experts to cover all different areas treated in the manuscript, which especially relevant in multidisciplinary research —I don’t understand how we expect two anonymous reviewers to assess a paper co-authored by a multidisciplinary team of ten or more authors!
Reagarding your other questions, I think we cannot give definite answers at the moment. Open peer review needs to be tried to prove its efficiency and most importantly its capacity for developing a new culture of collaboration and ethical conduct among scholars. For the moment it is an experiment, but a worthy one for our academic community to perform with seriousness and with a sense of responsibility before science and society.
A thoughtful piece on open peer review guidlines, with excellent timing, as it’s informed our discussion at Web Writing, a book in-progress which launched its six-week window for commentary yesterday. http://webwriting.trincoll.edu/how-this-book-evolved/process/#comment-7386
Thanks for your comments Panderis,
Indeed, conducting open review was *an experience*. I can see the benefits and I am also aware of the implications.
For instance, I am not totally convinced of the “neutrality” and “efficacy” of the “community” in providing feedback, just because they “are” in an open environment. This because we would need to define who the community would be. Independently of interacting online or face to face, people tend to gravitate towards those with whom they agree and/or those with whom they feel “comfortable”. I’ve seen examples of “community review” that are also biased…maybe not by “punishing” the author, but rather by providing rosy comments that do not take the debate of a paper forward either. So, I guess what I am trying to say is that power structures are always present. How we solve such conflict of interests… that’s the key. And that, for me, has to do with individual integrity and the values people share.
Having said that, I think openness can be (is) good. It does make us accountable in more visible ways. Yet, the quality of a paper review is only as good as its reviewers.
I’m not totally sure of the value of having someone only reviewing a given section of a paper though, as it is the sums of all parts – the paper as a whole – that will have merit or not. Maybe what we need in such cases is reviewers who are versed in multidisciplinary research.
As a final comment, I’d like to say that I do appreciate that open peer review is still at its inception stage. And I agree that it is an experiment that needs to be conducted so that we can advance our thinking and develop alternatives ways as to how research is assessed in a world where knowledge networks are more open and visible than ever. However, I want to believe that the academic community already performs in a serious and responsible fashion. What I think the open peer review brings is a new, different layer of reading and commenting on someone else’s work. And I hope that this “experiment” can take the practice of research reviewing to a new level; one of dialogue and discussion about a paper submitted for publication. This would be beneficial to all, and particularly to early career researchers, at whom the Journal of Perspectives in Applied Academic Practice (JPAAP) is aimed.