James Hartley argues that new technology used for submitting papers to academic journals increases the possibilities for gathering data, analysing it and improving the refereeing process.

My recent article (Hartley, 2012) on ‘Refereeing articles in the information age’ appears to have attracted some attention. I began by summarising some of the research on peer reviewing carried out before the advent of new technology (such as Scholar One) in this respect.  I then continued to list the pros and cons of this new technology – for authors, editors and publishers.  I showed, with an admittedly non-representative sample of 10 editors, that editors generally found the new technology to be more advantageous the larger the enterprise.

I also noted that much more information is now available to editors (and researchers) about the performances of authors, referees and editors.  One or two (again unrepresentative) articles suggest that older more experienced referees are less effective than younger more inexperienced ones, and that referees can be classified in terms of how critical they are. Similarly, records can be maintained about individual authors, as well as about house-keeping issues such as the number of submissions/acceptances per month, impact factors, and the like.

With new technology private comments between referees and editors can be made available for study.  The editors of Environmental Microbiology showed in 2010 that private comments made by referees range from the generous:

I very much enjoyed reading this paper, and do not have any significant comments. Wish I had thought of this one!

It is always a joy to review manuscripts such as this. Well-conceived, well executed, well edited. Clean, pristine from start to finish.

Through the mild:

It is sad to see so much enthusiasm and effort go into analysing a dataset that is just not big enough.

to the hostile:

The lack of negative controls in this study… results in the authors being lost in the funhouse.

The presentation is of a standard I would reject from an undergraduate…

However, there are few if any research studies on comments like these, and their effects on editorial decision-making are unknown.

I concluded my article by describing various arrangements (from open to ‘blind’ peer reviewing) where such comments can be made public (or not).  Four of these arrangements are:

                    Papers                                     Referees

1. Authors and affiliation deleted            Anonymous referees

2. Authors and affiliation included          Anonymous referees

3. Authors and affiliation included          Referees volunteer (or not) their names

4. Authors and affiliation included          Referees are named

There are arguments for and against all of these pairings.  In System 1, as far as the anonymity of authors is concerned, the evidence shows that referees can often identify the authors of papers, even though steps have been taken to prevent this.  With Systems 2, 3 and 4, if the authors are not already known to the referees, it is fairly easy to look them up on Google Scholar, for example, and to assess their track record – which might or might not bias a review.

Most journals operate either Systems 1 or 2.  Applied Psychological Measurement, for example, currently operates System 1, the British Journal of Educational Technology System 2, the Journal of Educational Psychology and PloS Medicine System 3, and the British Medical Journal, System 4.

System 4 is avoided by most journals, presumably because of the hassles of dealing with abuse and recriminations.  System 3 might avoid or lessen this problem.  However, a recent opinion poll suggested that 75% of referees preferred System 1 and that 60% said that they would be less likely to referee papers if their signed reports were published (Elsevier, 2010). 

Conclusions: a personal view

In this blog I have outlined some of the current practices used by editors, authors and referees when using electronic submission and publishing systems. Some of these practices are more open than others, but I believe, in this information age of WikiLeaks, Facebook and Twitter, that little – if anything – should be hidden from the different contributors to the total system. Thus I feel that it is System 4 that we should be aiming for when it comes to refereeing. What little evidence there is (as opposed to opinion) suggests that, with open refereeing, there will some improvement in the quality of the reports received, and an increase in the number of reviewers recommending publication, but that there will be an decrease in the number of reviewers willing to review. (See Bingham et al., 1998; Smith, 1999; van Rooyen et al, 1999; and Walsh et al, 2000). Open refereeing is controversial but, in this information age, it may be more appropriate.  

All of the citations in this blog can be found in:
Hartley, J.  (2012). Refereeing academic articles in the information age.  British Journal of Educational Technology, 43, 3, 520-528.

Note: This article gives the views of the author(s), and not the position of the Impact of Social Sciences blog, nor of the London School of Economics

Print Friendly