The h-index attempts to measure the productivity and impact of the published work of scholar. But reducing scholarly work to a number in this way has significant limitations. Stacy Konkiel highlights four specific reasons the h-index fails to capture a complete picture of research impact. Furthermore, there are a variety of new altmetrics tools out there focusing on how to measure the influence of all of a researcher’s outputs, not just their papers.
You’re surfing the research literature on your lunch break and find an unfamiliar author listed on a great new publication. How do you size them up in a snap? Google Scholar is an obvious first step. You type their name in, find their profile, and–ah, there it is! Their h-index, right at the top. Now you know their quality as a scholar.
Or do you?
The h-index is an attempt to sum up a scholar in a single number that balances productivity and impact. Anna, our example, has an h-index of 25 because she has 25 papers that have each received at least 25 citations. Today, this number is used for both informal evaluation (like sizing up colleagues) and formal evaluation (like tenure and promotion).
We think that’s a problem. The h-index is failing on the job, and here’s how:
1. Comparing h-indices is comparing apples and oranges.
Let’s revisit Anna LLobet, our example. Her h-index is 25. Is that good?
Well, “good” depends on several variables. First, what is her field of study? What’s considered “good” in Clinical Medicine (84) is different than what is considered “good” in Mathematics (19). Some fields simply publish and cite more than others.
Next, how far along is Anna in her career? Junior researchers have a h-index disadvantage. Their h-index can only be as high as the number of papers they have published, even if each paper is highly cited. If she is only 9 years into her career, Anna will not have published as many papers as someone who has been in the field 35 years.
Furthermore, citations take years to accumulate. The consequence is that the h-index doesn’t have much discriminatory power for young scholars, and can’t be used to compare researchers at different stages of their careers. To compare Anna to a more senior researcher would be like comparing apples and oranges.
Did you know that Anna also has more than one h-index? Her h-index (and yours) depends on which database you are looking at, because citation counts differ from database to database. (Which one should she list on her CV? The highest one, of course. :))
2. The h-index ignores science that isn’t shaped like an article.
What if you work in a field that values patents over publications, like chemistry? Sorry, only articles count toward your h-index. Same thing goes for software, blog posts, or other types of “non-traditional” scholarly outputs (and even one you’d consider “traditional”: books).
Similarly, the h-index only uses citations to your work that come from journal articles, written by other scholars. Your h-index can’t capture if you’ve had tremendous influence on public policy or in improving global health outcomes. That doesn’t seem smart.
3. A scholar’s impact can’t be summed up with a single number.
We’ve seen from the journal impact factor that single-number impact indicators can encourage lazy evaluation. At the scariest times in your career–when you are going up for tenure or promotion, for instance–do you really want to encourage that? Of course not. You want your evaluators to see all of the ways you’ve made an impact in your field. Your contributions are too many and too varied to be summed up in a single number. Researchers in some fields are rejecting the h-index for this very reason.
So, why judge Anna by her h-index alone?
Questions of completeness aside, the h-index might not measure the right things for your needs. Its particular balance of quantity versus influence can miss the point of what you care about. For some people, that might be a single hit paper, popular with both other scholars and the public. (This article on the “Big Food” industry and its global health effects is a good example.) Others might care more about how often their many, rarely cited papers are used often by practitioners (like those by CG Bremner, who studied Barrett Syndrome, a lesser known relative of gastroesophageal reflux disease). When evaluating others, the metrics you’re using should get at the root of what you’re trying to understand about their impact.
4. The h-index is dumb when it comes to authorship.
Some physicists are one of a thousand authors on a single paper. Should their fractional authorship weigh equally with your single-author paper? The h-index doesn’t take that into consideration.
What if you are first author on a paper? (Or last author, if that’s the way you indicate lead authorship in your field.) Shouldn’t citations to that paper weigh more for you than it does your co-authors, since you had a larger influence on the development of that publication?
The h-index doesn’t account for these nuances. So, how should we use the h-index? Many have attempted to fix the h-index weaknesses with various computational models that, for example, reward highly-cited papers, correct for career length, rank authors’ papers against other papers published in the same year and source, or count just the average citations of the most high-impact “core” of an author’s work.
None of these have been widely adopted, and all of them boil down a scientist’s career to a single number that only measures one type of impact. What we need is more data. Altmetrics–new measures of how scholarship is recommended, cited, saved, viewed, and discussed online–are just the solution. Altmetrics measure the influence of all of a researcher’s outputs, not just their papers. A variety of new altmetrics tools can help you get a more complete picture of others’ research impact, beyond the h-index. You can also use these tools to communicate your own, more complete impact story to others.
So what should you do when you run into an h-index? Have fun looking if you are curious, but don’t take the h-index too seriously.
This piece originally appeared on the Impactstory blog and is reposted with the author’s permission.
Note: This article gives the views of the author, and not the position of the Impact of Social Science blog, nor of the London School of Economics. Please review our Comments Policy if you have any concerns on posting a comment below.
Stacy Konkiel is the Director of Marketing & Research at Impactstory, an open-source, web-based tool that helps researchers explore and share the diverse impacts of all their research products. A former academic librarian, Stacy has written and spoken most often about the potential for altmetrics in academic libraries.
Stacy has been an advocate for Open Scholarship since the beginning of her career, but credits her time at Public Library of Science (PLOS) with sparking her interest in altmetrics and other revolutions in scientific communication. You can connect with Stacy on Twitter at @skonkiel.
I’ve used h-indices in hiring decisions and they’re sometimes helpful. Most people don’t believe that they’re useful on their own and the author of this post, who is promoting an alternative product (altmetrics) sets the h-index up as a straw person in her argument in favor of ‘holistic’ measures. I’d never use altmetrics as part of a hiring decision, though, because they are so easily gamed. I know this because gaming altmetrics is an important part of the impact strategy for papers from several of the groups that we work with.
Thanks for your perspective! Gaming is a common concern that’s raised with respect to altmetrics, and it’s true that altmetrics–like all quantitative measures of impact–can be gamed. Altmetrics is no different from citations in that respect, and we hear often of “I’ll scratch your back, you scratch mine” citation swaps and rings in academia. It’s an unfortunate side effect of a system that relies heavily on quantitative measures for understanding the impact of work.
But! What’s great about altmetrics–which are quick to accumulate and in many cases sourced from more transparent data sources that citations–is that gaming is more easily sniffed out. For example, with a single click, I can scan the contents of tweets and realize that they’re coming from bogus accounts, or that they all just repeat the headline of the article that’s being shared, rather than adding value that proves the tweeter read and understands the article. Or, in the case of downloads, many systems have adapted to detect gamers and will remove downloads that occur from particular IP addresses (SSRN is perhaps the most famous example). For citations, on the other hand, it can be a more onerous task to detect the sort of gaming that occurs in old boys networks (you cite my paper, I’ll cite yours).
All of that aside, I’m curious as to what your perspectives are on the points raised in the post–beyond altmetrics vs. h-indices. Does your field consider alternative research outputs (like software and datasets) in hiring? And what about the issue of authorship, particularly the case of many authors on a single paper?
What did you find helpful about using an h-index in hiring?
Yes, there are problems with the h-index. But many of those cited in the posting can be fixed. There is a Public or Perish (PoP) equivalent that canvasses book and book chapter citations, as well us unpublished reports and journals not covered in the conventional h-index. And then you can adjust the h-index for the length of a person’s career, how recent the publications were, how many authors etc. So, all these difficulties are fixable. But of course the remaining concern is the partial nature of the index – however derived – and for that there is a simple solution: do not use any bibliometric measure on its own. It is a powerful insight nevertheless.
I think the only sound reason here is #4. #1 is functionally irrelevant because we are not really compared to people outside our own area, or at dissimilar stages. The h-index becomes relevant on job searches, for instance, and most often you’ll be compared with people at roughly the same stage in the same field. Even for many grants this is true. #2 and #3 are functionally irrelevant because the h-index is presented *with* a CV. I’ve never seen an h-index by itself as an argument for anything.
#4 is true, but since authorship “rules” (traditions?) are different even across different labs within the same field — and are often implicit — it is difficult to see how any metric can take that into account.
The H-Index also does not distinguish self-citations from citations by others, and also does not distinguish negative citations from positive ones.
The number of citations a researcher/author receives tells something about their visibility. I believe it is a good thing that the place on the paper does not count, it is the contribution that counts. Not using metrics at all at any point puts scientists passionate for their work and visible for it in the shadow. It opens all the doors again for politics and corruption. You may realize why metrics have been introduced in the first place when you are not lobbying and licking heels all the time like a crazy cat. It is just another piece of information, like any publication is composed of many pieces of information, and of course everyone knows it is not the complete story. So no reason to be all negative about it, on the contrary as a woman I would fully embrace it as one of the parameters for self- and others evaluation.