Peer review may be favoured as the best measure of scientific assessment ahead of the REF, but can it be properly implemented? Peter Coles does the maths on what the Physics panel face and finds there simply won’t be enough time to do what the REF administrators claim. Rather, closed-access bibliometrics will have to be substituted at the expense of legitimate assessment of outputs.
What I want to do first of all is to draw attention to a very nice blog post by a certain Professor Moriarty who, in case you did not realise it, dragged himself away from his hiding place beneath the Reichenbach Falls and started a new life as Professor of Physics at Nottingham University. Phil Moriarty’s piece basically argues that the only way to really judge the quality of a scientific publication is not by looking at where it is published, but by peer review (i.e. by getting knowledgeable people to read it). This isn’t a controversial point of view, but it does run counter to the current mania for dubious bibliometric indicators, such as journal impact factors and citation counts.
The forthcoming Research Excellence Framework involves an assessment of the research that has been carried out in UK universities over the past five years or so, and a major part of the REF will be the assessment of up to four “outputs” submitted by research-active members of staff over the relevant period (from 2008 to 2013). Reading Phil’s piece might persuade you to be happy that the assessment of the research outputs involved in the REF will be primarily based on peer review. If you are then I suggest you read on because, as I have blogged about before, although peer review is fine in principle, the way that it will be implemented as part of the REF has me deeply worried.
The first problem arises from the scale of the task facing members of the panel undertaking this assessment. Each research active member of staff is requested to submit four research publications (“outputs”) to the panel, and we are told that each of these will be read by at least two panel members. The panel comprises 20 members.
As a rough guess let’s assume that the UK has about 40 Physics departments, and the average number of research-active staff in each is probably about 40. That gives about 1600 individuals for the REF. Actually the number of category A staff submitted to the 2008 RAE was 1,685.57 FTE (Full-Time Equivalent), pretty close to this figure. At 4 outputs per person that gives 6400 papers to be read. We’re told that each will be read by at least two members of the panel, so that gives an overall job size of 12800 paper-readings. There is some uncertainty in these figures because (a) there is plenty of evidence that departments are going to be more selective in who is entered than was the case in 2008 and (b) some departments have increased their staff numbers significantly since 2008. These two factors work in opposite directions so not knowing the size of either it seems sensible to go with the numbers from the previous round for the purposes of my argument.
There are 20 members of the panel so 6400 papers submitted means that, between 29th November 2013 (the deadline for submissions) and the announcement of the results in December 2014 each member of the panel will have to have read 640 research papers. That’s an average of about two a day…
It is therefore blindingly obvious that whatever the panel does do will not be a thorough peer review of each paper, equivalent to refereeing it for publication in a journal. The panel members simply won’t have the time to do what the REF administrators claim they will do. We will be lucky if they manage a quick skim of each paper before moving on. In other words, it’s a sham.
Now we are also told the panel will use their expert judgment to decide which outputs belong to the following categories:
- 4* World Leading
- 3* Internationally Excellent
- 2* Internationally Recognized
- 1* Nationally Recognized
- U Unclassified
There is an expectation that the so-called QR funding allocated as a result of the 2013 REF will be heavily weighted towards 4*, with perhaps a small allocation to 3* and probably nothing at all for lower grades. The word on the street is that the weighting for 4* will be 9 and that for 3* only 1. “Internationally recognized” will be regarded as worthless in the view of HEFCE. Will the papers belonging to the category “Not really understood by the panel member” suffer the same fate?
The panel members will apparently know enough about every single one of the papers they are going to read in order to place them into one of the above categories, especially the crucial ones “world-leading” or “internationally excellent”, both of which are obviously defined in a completely transparent and objective manner. Not. The steep increase in weighting between 3* and 4* means that this judgment could mean a drop of funding that could spell closure for a department.
We are told that after forming this judgement based on their expertise the panel members will “check” the citation information for the papers. This will be done using the SCOPUS service provided (no doubt at considerable cost) by Elsevier, which by sheer coincidence also happens to be a purveyor of ridiculously overpriced academic journals. No doubt Elsevier are on a nice little earner peddling meaningless data for the HECFE bean-counters, but I have no confidence that they will add any value to the assessment process.
There have been high-profile statements to the effect that the REF will take no account of where the relevant “outputs” are published, including a pronouncement by David Willetts. On the face of it, that would suggest that a paper published in the spirit of Open Access in a free archive would not be disadvantaged. However, I very much doubt that will be the case.
I think if you look at the volume of work facing the REF panel members it’s pretty clear that citation statistics will be much more important for the Physics panel than we’ve been led to believe. The panel simply won’t have the time or the breadth of understanding to do an in-depth assessment of every paper, so will inevitably in many cases be led by bibliometric information. The fact that SCOPUS doesn’t cover the arXiv means that citation information will be entirely missing from papers just published there.
The involvement of a company like Elsevier in this system just demonstrates the extent to which the machinery of research assessment is driven by the academic publishing industry. The REF is now pretty much the only reason why we have to use traditional journals. It would be better for research, better for public accountability and better economically if we all published our research free of charge in open archives. It wouldn’t be good for academic publishing houses, however, so they’re naturally very keen to keep things just the way they are. The saddest thing is that we’re all so cowed by the system that we see no alternative but to participate in this scam.
Incidentally we were told before the 2008 Research Assessment Exercise that citation data would emphatically not be used; we were also told afterwards that citation data had been used by the Physics panel. That’s just one of the reasons why I’m very sceptical about the veracity of some of the pronouncements coming out from the REF establishment. Who knows what they actually do behind closed doors? All the documentation is shredded after the results are published. Who can trust such a system?
To put it bluntly, the apparatus of research assessment has done what most bureaucracies eventually do; it has become entirely self-serving. It is imposing increasingly ridiculous administrative burdens on researchers, inventing increasingly arbitrary assessment criteria and wasting increasing amounts of money on red tape which should actually be going to fund research.
And that’s all just about “outputs”. I haven’t even started on “impact”….
This was originally posted on Peter Coles’ personal blog and is reposted with permission.
Note: This article gives the views of the author, and not the position of the Impact of Social Science blog, nor of the London School of Economics.
Peter Coles is Professor of Theoretical Astrophysics and Head of the School of Mathematical and Physical Sciences at the University of Sussex. His research is in the area of cosmology and the large-scale structure of the Universe.
I think it makes much more sense to rely on the peer review already done by journals and the post-review done by other researchers citing papers than to do a second very cursory peer review by the REF panel. I was a peer-reviewer for the ERA 2012 here in Australia and I agree that it is really hard to make judgements on a whole pile of articles many not really in your area of expertise in a short amount of time. I just think this is totally a waste of resources when peer review has already been carried out on these articles. Instead bibliometrics should be used even more extensively if governments want to conduct these kind of assessment exercises. However, I agree that the weighting given to REF outcomes in funding decisions in the UK seems very extreme.
The problem is, the REF wants to review for quality of work, whereas journals review largely for sexiness. The qualities that allow a work to make a splash in a high-profit journal are not necessarily those that would make you want to reward the responsible institution going forward.
METRICS ARE A SUPPLEMENT, NOT A SUBSTITUTE FOR PEER REVIEW
1. REF submissions are already peer-reviewed.
2. The journal impact factor is not the only metric of importance and impact.
3. Open Access (OA) will generate many rich new metrics of importance and impact.
4. But articles have to be OA first, before we can have those rich new metrics.
5. Yes, re-peer-reviewing peer-reviewed research is a waste of time and money.
6. But metrics can supplement peer review and have been showed to be predictive of peer rankings.
7. HEFCE’s proposal to mandate immediate-deposit in institutional repositories a condition for eligibility for REF will help make RCUK OA policy effective.
8. OA in turn will generate the rich new metrics that will make REF less time-consuming and expensive.
Harnad, S. (2003) Measuring and Maximising UK Research Impact. Times Higher Education Supplement. Friday, June 6 2003
Harnad, S., Carr, L., Brody, T. & Oppenheim, C. (2003) Mandated online RAE CVs Linked to University Eprint Archives: Improving the UK Research Assessment Exercise whilst making it cheaper and easier. Ariadne 35.
Brody, T., Carr, L., Harnad, S. and Swan, A. (2007) Time to Convert to Metrics. Research Fortnight pp. 17-18.
Brody, T., Carr, L., Gingras, Y., Hajjem, C., Harnad, S. and Swan, A. (2007) Incentivizing the Open Access Research Web: Publication-Archiving, Data-Archiving and Scientometrics. CTWatch Quarterly 3(3).
Harnad, S. (2008) Validating Research Performance Metrics Against Peer Rankings. Ethics in Science and Environmental Politics 8 (11) doi:10.3354/esep00088 The Use And Misuse Of Bibliometric Indices In Evaluating Scholarly Performance
Harnad, S. (2009) Open Access Scientometrics and the UK Research Assessment Exercise. Scientometrics 79 (1) Also in Proceedings of 11th Annual Meeting of the International Society for Scientometrics and Informetrics 11(1), pp. 27-33, Madrid, Spain. Torres-Salinas, D. and Moed, H. F., Eds. (2007)
“All the documentation is shredded after the results are published.”
How bizarre. Do you have any idea why this is?
HEFCE/REF Immediate-Deposit Proposal to the Recue
I think this is a hold-over from the absurd time when HEFCE paid publishers for the right to have a print copy of the submissions, and then the payment was waived on condition that the copies were destroyed after the exercise.
But if the new HEFCE/REF mandate proposal is adopted, all submissions will be authors’ final peer-reviewed drafts, deposited in their institutional repositories immediately upon acceptance for publication. — So, nothing to shred.
(At least 60% of the deposits will be immediately OA; for the immediate-deposits that are embargoed, there is the facilitated email eprint request Button, requiring one click from the requester and then one click from the author, that will tide over user needs during any embargo. This is not yet quite Green OA, but Almost-OA. And of course all sensible authors will make their deposits immediately OA…)
http://www.eprints.org/news/features/request_button.php
https://wiki.duraspace.org/display/DSPACE/RequestCopy
Sale, A., Couture, M., Rodrigues, E., Carr, L. and Harnad, S. (2012) Open Access Mandates and the “Fair Dealing” Button. In: Dynamic Fair Dealing: Creating Canadian Culture Online (Rosemary J. Coombe & Darren Wershler, Eds.)
I suspect another reason for destruction of all records was/is to avoid lengthy appeals against decisions taken by REF/RAE panels.
The claim that citation counts are dubious is nonsense, as numerous studies have shown that citation counts correlate strongly with all other measures of peer esteem – and in all subject areas. The comment that SCOPUS does not cover Arxiv is partly incorrect. If someone cites an Arxiv item in a journal article, then this gets added to the SCOPUS database. You are correct in stating that Elsevier is charging HEFCE for access to SCOPUS for the REF. I was on the committee that decided to award the contract to Elsevier, and have to say it represented the best value for money of the short-listed tenders.
“The claim that citation counts are dubious is nonsense, as numerous studies have shown that citation counts correlate strongly with all other measures of peer esteem – and in all subject areas.”
These would be very interesting to see. Do you have references?
Start with my articles, then look (using SCOPUS or Web of Knowledge or Google Scholar) at who has cited them, then do a general search on “research assessment exercise” or “RAE” and “citation”. My articles include:
C. Oppenheim, The correlation between citation counts and the 1992 Research Assessment Exercise Ratings for British library and information science university departments, J. Doc., 1995, 51 (1), 18-27.
C. Oppenheim, Do citations count? Citation indexing and the Research Assessment Exercise, Serials, 1996, 9, 155-161.
C. Oppenheim, The correlation between citation counts and the 1992 Research Assessment Exercise ratings for British research in genetics, anatomy and archaeology, Journal of Documentation, 1997, 53 (5), 477-487.
A. Holmes and C. Oppenheim, Use of citation analysis to predict the outcome of the 2001 RAE for Unit of Assessment 61: Library and Information Management, Information Research, 2001, 6 (2), http://www.shef.ac.uk/~is/publications/infres/6-2/paper103.html.
V. Bence and C. Oppenheim, Journals, scholarly communication and the RAE: a case study of the business and management sector, Serials, 2001, 14 (3), 265-274.
M. Norris and C. Oppenheim, Citation counts and the Research Assessment Exercise V: Archaeology and the 2001 RAE, Journal of Documentation, 2003, 59 (6), 709 – 730.
C. Oppenheim and M. Summers, Citation counts and the RAE VI: Unit of Assessment 67 (music), Information Research, 2008, 13 (2), http://www.informationr.net/ir/13_2/paper342.html
Thanks, that’s very comprehensive.
My feeling on assessments such as the RAE and the REF — and it’s only a feeling, not backed by data — that the use of article-level bibliometrics such as citation count, while imperfect, is at least an attempt to measure the right thing; whereas the use of journal-level bibliometrics such as impact factor are completely, completely wrong.
Totally agree. I am opposed to the use of IF to evaluate research. Likewise I am opposed to HEIs encouraging or requiring staff to only submit to journals with a high IF.
Charles,
Did the RAEs of, e.g., 1992 or 2001, not suffer from the same problem as suggested by the article? i.e., panelists relying on citation counts, instead of peer-reviewing the submissions? If so, wouldn’t this explain the correlations in your papers?