Why does the impact factor continue to play such a consequential role in academia? Alex Rushforth and Sarah de Rijcke look at how considerations of the metric enter in from early stages of research planning to the later stages of publication. Even with initiatives against the use of impact factors, scientists themselves will likely err on the side of caution and continue to provide their scores on applications for funding and promotion.
This piece is part of a series on the Accelerated Academy.
A number of criticisms have emerged in recent times decrying the Journal Impact Factor (JIF) and the perverse effects this indicator is having on biomedical research. One notable feature of these disgruntled statements is they have often emerged from within the medical research communities themselves. For instance it is now commonplace to read denunciations of metric cultures in the editorial statements of eminent medical journals, or bottom-up movements by professional scientists to protest and reform the governance of science.
Despite these rumours, how the Journal Impact Factor intersect with knowledge making practices of researchers on the ‘shop floor’ in biomedicine is little known. Without detailed empirical studies it is difficult to foretell of the actual and possible consequences this indicator is having for biomedical knowledge work. For this reason we set out to study this topic in a project we recently completed entitled ‘the impact of indicators’. We focussed on three kinds of research groups in two University Medical Centres (academic medical schools) in the Netherlands. Our ethnographic approach looked at groups in basic, translational, and applied areas of biomedicine, a set of distinctions commonly made within the field. These sub-cultures all exhibit different ways of making knowledge (Knorr-Cetina, 1999), and their varying interactions with this indicator in the course of their day-to-day research practices was the focus of our analysis. This included observing how the JIF entered from early stages of research planning and collaboration to the later stages of publication.
Image credit: moise_theodor (CC 0)
The lab-based groups in the basic and translational areas exhibited particularly tight coupling with the JIF as de facto standard for measuring quality and novelty of their work. For this reason we will briefly outline some of the most striking impressions we got from studying these particular kinds of groups, focusing particularly on the final stages of the knowledge production process i.e. publishing (for a more detailed account of the production process in all three biomedical areas see Rushforth and de Rijcke, 2015).
Whilst decisions and practices scientists make about their work remain multi-dimensional, the JIF has become a kind of obligatory passage point though which other kinds of considerations need to be filtered. For instance, basic scholarly decisions such as the types of audience one would like to reach with a paper, whom one should collaborate with, and how much time and resources should be dedicated to performing extra experiments, are all weighed up against the likelihood it will land their end-product (i.e. a journal article) in a higher impact journal. Such is the conviction that this is the currency through which they will be received within their own group, peer networks, department, by funding agencies and indeed the academic job market, our informants could pursue this criteria somewhat religiously. In one lab we observed a debate about whether to submit a manuscript to a particular journal based on the fact it scored 0.1 of a decimal point higher than the other journal being considered.
As intelligent and reflective individuals what did informants make of being caught up in this process? Scientists must accommodate demands to publish high impact work in order to get on and have a career. There is little way around this. At the same as fulfilling these demands they recognise ‘realities being knowingly eclipsed’ (Strathern, 2000). Some were wise to the wider criticisms of the indicator in circulation, be they technical or more sociological. Some decried the fact they and colleagues were making arbitrary decisions about where to submit manuscripts based on what was effectively a rather crude ranking exercise. Others though offered tentative defences of the impact factor’s effects. Indeed it is important not to paint the scientists as simply passive victims here. Despite its arbitrariness, the JIF does offer those who play by the rules of the game the pleasures of narcissism (Roberts, 2009) and play (Graeber, 2015). But whether critical or supportive, what’s for sure is there could be no stepping away.
So what might the findings from this exploratory study signal?
An important set of concerns one takes from studying the JIF’s effects is the likely consequences all this has for biomedical knowledge production. Unfortunately as a recent literature review with our colleagues on the effects of indicators showed, this is largely impossible to answer in any general definitive sense (De Rijcke et al., 2015). Nonetheless, there are a few factors we might wish to flag.
In respect to research evaluation, whilst there are temptations in using the JIF as a short-cut to making judgements about the quality of work, there are several reasons to be concerned about increasing reliance on this indicator alone. The fact is that by deferring to the JIF, one is by extension deferring the complex process of judging and ranking quality towards second order criteria (how well cited the journal is in which the article appears), rather than the merits of the work being judged as a standalone contribution to knowledge. This not only reduces the worth of scientific work to a simple numbers game, but also rely solely on the peer review process of journals to determine scientific good taste. Whereas we might generally trust journals to do this well enough most of the time, even the big brand journals have been exposed for failing to detect simple errors or misconduct, as Philip Moriarty’s recent keynote at the Accelerated Academy workshop in Prague made evident. It is also commonly argued in history and sociology of science that journal peer review also errs towards conservatism, rejecting radical innovations and erring more towards incremental, normal science-type contributions. Our observations of biomedical researchers suggests charges of conservatism made against journal peer review and reinforced by the JIF, should be taken as more than simply sour grapes from those who have work rejected by high impact journals: given the high stakes that come with publishing in high impact titles, researchers are being steered towards exercising great caution in their decisions about which projects to pursue. If there’s not chance an idea will land them the high impact outputs they need, it’s a non-starter.
Depressingly this trend looks set to continue, as long as there is hyper-specialisation and pressures to be productive, those evaluating grant applications and hiring and promotion decisions may not feel equipped in terms of skills or resources to read through submissions and make their own judgements. Evaluation committees may be pressured into omitting impact factors from their considerations, however their easy availability online, forces of habit, and discretion entrusted to expert committees makes this difficult to police. Even if such initiatives are being made, scientists themselves will likely err on the side of caution and continue to provide their scores on applications for funding and promotion. The appeal of the JIF, or some equivalent, thus looks likely to endure in a number of areas of medicine until this cycle is broken-up.
The post is part of a series on the Accelerated Academy and is based on the author’s contribution presented at Power, Acceleration and Metrics in Academic Life (2 – 4 December 2015, Prague) which was supported by Strategy AV21 – The Czech Academy of Sciences. Videocasts of the conference can be found on the Sociological Review.
Note: This article gives the views of the author, and not the position of the LSE Impact blog, nor of the London School of Economics. Please review our Comments Policy if you have any concerns on posting a comment below.
Alex Rushforth is a researcher at CWTS, Leiden University. He works in the areas of sociology of science, higher education, and organizational sociology. Whilst training at the Universities of Surrey and York in the United Kingdom, he has developed interests in the evolving governance of public sciences, and in particular its impact upon the research process itself.
Sarah de Rijcke is a sociologist of science and technology at CWTS, where she leads the Evaluation Practices in Context (EPIC) research group. Her current research programme in evaluation studies examines interactions between research assessment and practices of knowledge production. This programme is situated at the intersection of Science and Technology Studies and the sociology and anthropology of science.
The citation rates of the individual papers are also easily available online. Even if the IF were a valid measure of journal quality, and it is not, why would anyone choose to use it instead of the individual papers’ actual citation rate?
Actually, in terms of methodological quality, the data is rather clear: the higher the JIF, the lower the quality:
http://journal.frontiersin.org/article/10.3389/fnhum.2013.00291/full
So using this metric to, i.e. base one’s work on, this work is more likely to fail. If you use it to, e.g., select scientists for hiring/promotion, you’ll more likely higher people producing unreliable work.
Most Nature authors benefit from an IF generated by a happy few.
https://bernardrentier.wordpress.com/2015/12/31/denouncing-the-imposter-factor/
I totally agree with the comment of Lozano. The citation rate of individual articles is a much better indicator of the quality of an article than the impact factor of the journal, in which it has been published. Furthermore, the citations of articles correlate only moderately with journal impct factors. I think that journal impact factors might have been more important in the days researchers mainly cited articles in journals they subscribed to themselves. These days, with computer seachers and digital journals, researchers often find relevant articles in journals they have never heard of.
I cannot see any problem with a researcher or research team aspiring to publish their work in a “good” journal. Often it comes down to a small group of candidate journals, and it is surely logical to select the journal that is most likely to be highly regarded and cited. This is not an academic pathology; it is sensible scientific practice. If the JIF is so bad as everybody makes out, what alternative are people suggesting to assist in selecting a “good” journal? In a world of thousands of journals, millions of articles, and many hundreds of academics seeking promotion and going for new jobs, I cannot see how one can get away from using metrics in a judicious manner.
Peter Davis wrote:
“I cannot see any problem with a researcher or research team aspiring to publish their work in a “good” journal.”
Then the choice is determined what people call a “good journal”. For me, any journal that hides my work behind a paywall is a bad journal, for instance.
“it is surely logical to select the journal that is most likely to be highly regarded and cited.”
If the data show that “highly regarded” is correlated with “likely to be wrong”, I cannot see anything logical in such a selection. Likewise, if the data show that it is the articles that make the journal highly cited and not the other way around, your contention is even abjectly illogical.
“If the JIF is so bad as everybody makes out, what alternative are people suggesting to assist in selecting a “good” journal?”
If you go by the data, then throwing dice is better than using JIF, as it is at least neutral with respect to methodological quality. In fact, the differences between journals in terms of methodological quality are so small that random chance is likely also the most rational way of selecting one above others.
As we scientists are also humans, data may not be the only factor driving a decision, however. One also has to factor in subjective psychological concerns such as the satisfaction felt with a decision. It is conceivable that the data-driven decision of using chance may not feel satisfactory for everyone.
In such a human case, ruling out subscription journals is a good strategy, as subscriptions decrease the number of possible readers. Moreover, ruling out journals which reject manuscript based on subjective criteria such as novelty or interest would also be smart, as resubmissions without actual suggestions for improvement only take time away from the next experiments.
If you follow these simple rules, you can probably narrow down the list of journals to a very manageable short list.
“In a world of thousands of journals, millions of articles, and many hundreds of academics seeking promotion and going for new jobs, I cannot see how one can get away from using metrics in a judicious manner.”
That is a completely different notion from every single sentence you wrote above this statement. The data show that using journal-level metrics is not judicious. So using metrics judiciously as you correctly suggest, rules out all the other explicit and implied suggestions in the rest of your comment.
The first step of using metrics judiciously is to use article-level metrics.