The Declaration on Research Assessment (DORA) has yet to achieve widespread institutional support in the UK. Elizabeth Gadd digs further into the slow uptake. Although there is growing acceptance that the Journal Impact Factor is subject to significant limitations, DORA feels rather negative in tone: an anti-journal metric tirade. There may be times when a journal metric, sensibly used, is the right tool for the job. By signing up to DORA, institutions may feel unable to use metrics at all.
The recent Metric Tide report recommended that institutions sign up to the San Francisco Declaration on Research Assessment (DORA). DORA was initiated by the American Society of Cell Biology and a group of other scholarly publishers and journal editors back in 2012 in order to “improve the ways in which the outputs of scientific research are evaluated”. Principally, it is a backlash against over-use of the Journal Impact Factor to measure the research performance of individual authors or individual papers, although its recommendations reach further than that. Subsequent to the publication of DORA, the bibliometric experts at CWTS in Leiden published the Leiden Manifesto (April 2015). This too is set against the “Impact Factor obsession” and offers “best practice in metrics-based research assessment so that researchers can hold evaluators to account, and evaluators can hold their indicators to account”. There is no option to sign up to this.
Having attended a couple of events recently at which it was highlighted (with something of a scowl) that only three UK HE institutions had signed up to DORA (Sussex, Manchester and UCL if you’re interested), I set about trying to ascertain where the UK HE community had got to in their thinking about this. I created a quick survey and advertised it on Lis-Bibliometrics, a forum for people who are interested in the use of bibliometrics in UK Universities with 533 members, and more recently to the Metrics Special Interest Group of the Association of Research Managers and Administrators (ARMA) with again over 500 members. Simply put, I was interested in whether institutions had signed, why or why not, what they saw as the pros and cons, and whether they were thinking about developing an internal set of principles for research evaluation – something else recommended by the Metric Tide Report. The survey was open between 9 September and 6 October 2015.
Image credit: Johnny Magnusson (Website) Public Domain
Perhaps rather tellingly, only 22 people responded. Twenty-two out of a potential 1000 – a response rate of about 2%. Now, there could have been a number of reasons for this: survey fatigue; not reaching the right audience; wrong time of year, etc., etc. However, as eight of the 22 openly declared that they’d not yet considered DORA (“don’t know what it is!”, stated one), I think it is more likely that the message hasn’t got out yet.
Of the remaining 16, three had already signed up to DORA and nine were actively considering it. Two other categories were offered (“actively considering and decided to sign” and “actively considering and decided not to sign”), but neither of these were selected. In subsequent free-text comments, two respondents mentioned that individuals within their organization had signed, and another pointed out that they were signatories by virtue of being members of LERU.
Figure 1 Which of the following best describes your institution?
The next question asked those that had decided to sign (or not) what their reasoning was. Although eight responses were received, only one really outlined their reasons: “It was about the principle(s), rather than over-evaluation of the precise wording. Also, it was about commitment to the direction of travel, rather than having to have everything in place before being able to sign up. Equally, signing up doesn’t mean we’re against using metrics; the opposite rather, that we’re wanting to use metrics, but in the right way(s).”
The follow-up question yielded more detail about the pros and cons of signing as the respondents saw them. There were nine responses. Amongst the pros were, “making a stand for responsible use of metrics” and stopping “the use of some metrics that are unhelpful”. Indeed one respondent felt that having raised the issue with senior managers had had a positive impact on “research assessment for academic promotion”, even though their institution hadn’t yet signed. Another hoped that by signing it would “keep administrators at bay that seek simplistic measures for evaluating complex issues”.
In terms of the cons, these were more varied. Three respondents were concerned that as a result of signing, their institution may feel unable to use metrics at all and, indeed, that the Journal Impact Factor may actually be a useful metric in the right circumstances. One respondent was concerned that signing DORA in and of itself would not bring about institutional culture change and wondered what the process would be for dealing with those that did not comply. The latter point may have been implicit in the response of two raising the issue of who in the institution would actually be responsible for signing (and therefore, presumably, responsible for monitoring compliance).
Figure 2: Who has been involved in the deliberations over whether to sign DORA?
A question about who had been involved in DORA deliberations at their institution showed that a wide variety of staff had done so. However, in the majority were senior University managers (10) followed by Library and Research Office staff (8). Interestingly there are no librarians among the original list of DORA signatories, although now at least 88 of the 12,522 total signatories have Library in their job title. (This figure may be slightly inflated as I spotted at least one signatory who was listed three times – perhaps they were a strong advocate?).
It was pleasing to see that academic staff had also played a part in at least some institutions’ DORA discussions, for when we talk about research evaluation it is them and their work that we are talking about. It was perhaps disappointing that academic staff weren’t involved more often. It was also interesting that in one case a Union representative had been involved. I think when you consider the implications of DORA from all these perspectives, it is not surprising that making a decision to sign is sometimes a long and drawn-out process. Whilst a research manager might view DORA as a set of general principles – a “direction of travel” – could you be certain that a Union Rep wouldn’t expect more from it?
As an example of the aforementioned research manager, I get the point of DORA – don’t use journal measures to measure things that aren’t journals, and especially try to avoid the Journal Impact Factor which is subject to significant limitations. I’ve no problem with that. However, I’m concerned that DORA could be mis-interpreted as a directive to avoid the use of all journal metrics in research evaluation as there are times when a journal metric, sensibly used, is the right tool for the job. Indeed in a ARMA Metrics SIG discussion around the DORA survey, Katie Evans, Research Analytics Librarian at the University of Bath provided a great list of scenarios in which journal metrics may be a useful tool in your toolkit. I have her permission to reproduce them here:
- Journal metrics gave the best correlation to REF scores (Metrics Tide, Supplementary Report II). This suggests that we might want to use them at a University/Departmental level. That might then filter down to use to inform assessment of individuals’ publishing records so that there’s consistency.
- Not all academic journals are equal in quality standards. Looking at the journal an item has been published can indicate that the work has met a certain standard. Journal metrics are an (imperfect) indicator of this.
- Article-level citation metrics are no use for recently published items that haven’t had time to accrue citations. Journal metrics are available straight away.
- For publishing strategy – a researcher has some control over where they publish; it’s a choice they can actively make to aim for a high quality journal. A researcher doesn’t have this sort of direct control over article-level citations. So you could argue that it’s fairer to ask someone to publish in journals of a certain standard than to meet article-level citation criteria.
For me, DORA feels rather negative in tone: an anti-journal metric tirade. One almost feels a bit sorry for Thomson Reuters (owners of the JIF) who, in my experience, always seem keen to set out both the value and limitations of bibliometrics in research evaluation. Personally I feel a lot more comfortable with the Leiden Manifesto which takes a more positive approach: a balanced call for the sensible, contextualized and transparent use of all publication metrics. If the Leiden Manifesto was available for signing, I wouldn’t hesitate. So, how about it Leiden?
The views expressed in this piece are my own and do not necessarily reflect those of my institution.
Note: This article gives the views of the author, and not the position of the Impact of Social Science blog, nor of the London School of Economics. Please review our Comments Policy if you have any concerns on posting a comment below.
Even if they were perfect, journal metrics ought to be used only to measure JOURNALS, not the papers therein. The article agrees with this point, but then goes on to point out four instances whereby journal metrics “might be useful”, four instances that specifically extend the use of journal metrics beyond their intended purpose.
“Journal metrics gave the best correlation to REF scores”.-
Perhaps, but the most widely used journal metric, the infamous impact factor, is also a good predictor of retraction rate. Although the use of impact metrics is specifically precluded (“No sub-panel will make any use of journal impact factors, rankings, lists or the perceived standing of publishers in assessing the quality of research outputs.”), biases might still ocur. One would have to examine the reasons behind the correlation between “journal metrics” and REF scores.
“Not all academic journals are equal in quality standards”.-
This is certainly true, but there is not need to look at the journal. Article-level metrics are readily available. Of course, it might be better to actually read the papers.
“Article-level citation metrics are no use for recently published items that haven’t had time to accrue citations. Journal metrics are available straight away.”
It is true that some good papers are not cited for years and then become citation classics, while other good papers get picked up right away. However, the vast majority of papers probably hit their peak citation rate in 3 to 5 years, depending on the field. In any case, journal metrics depend on PREVIOUS papers, not current ones. For the few situations in which recent papers have to be assessed to break a tie between 2 candidates, instead of trying to determine which journal metric is the most appropriate, it might actually be better just to pick up the 2 papers and read them.
“For publishing strategy – a researcher has some control over where they publish; it’s a choice they can actively make to aim for a high quality journal. A researcher doesn’t have this sort of direct control over article-level citations. So you could argue that it’s fairer to ask someone to publish in journals of a certain standard than to meet article-level citation criteria.”.-
The first problem here becomes defining “high quality”. Even if the data were to indicate that the prestige of elite journals is declining, public perceptions will take longer to change. Second, everyone probably aims as high as possible, but nobody really has “direct control” over which journal eventually publishes a given paper. Third, what is more informative, the opinion of an editor and 2 or 3 reviewers, or the opinion of the entire global scientific community who might choose to ignore or cite a given paper? Finally, what would be more impressive: a paper published in an elite journal that gets x citations over the next y years, or a paper published in a more modest journal but with the same citation history? A paper in an elite journal not only reaches more people, but it is immediately associated with excellence. If anything, there ought to be a correcting factor whereby a paper’s citation rate gets divided by the journal’s perceived prestige.