LSE - Small Logo
LSE - Small Logo

Blog Admin

September 17th, 2014

Proof over promise: Moving citation metric systems beyond journal impact towards a career impact approach.

8 comments

Estimated reading time: 5 minutes

Blog Admin

September 17th, 2014

Proof over promise: Moving citation metric systems beyond journal impact towards a career impact approach.

8 comments

Estimated reading time: 5 minutes

aw2013headRotterdam, 19-06-2007 RSM MijnhardtPublishing in a high-impact journal carries the implicit promise that the article will also be highly cited. But the proof of this logic remains unsubstantiated. By combining more accurate citation metrics, like the hIa-index and the citation-per-author-per-year metric, Anne-Wil Harzing and Wilfred Mijnhardt provide a more substantial alternative to the narrow journal-based metric. This combined metric provides a more reliable comparison between academics in different disciplines and at different career stages.

Economists love to rank. Even a casual five-minute literature search reveals literally hundreds of publications on rankings of academic productivity and impact. Dutch economists are no exception. In fact, they produced what, to our best knowledge, is the oldest ranking in the field: a nation-wide ranking of Economists (the Economics top-40) that has entered its fourth decade. However, this ranking is based on publication volume in ISI-listed journals weighted by the journal’s impact factor or article influence score, rather than on the actual impact of the publications in question.

In this blog we explain the added value of a career-impact focus and envision some practical implications for business schools.  We will show that an impact driven approach using Google Scholar creates a more democratic and inclusive assessment.

Proof (impact) and Promise (journals)

On average, publications in high-impact journals by definition get cited more frequently than publications in low-impact journals as the journal impact factor is based on average citations. This still is the fundamental logic behind the creation and maintenance of Journals list by business schools and associations such as ABS. We call this principle “promise”, i.e. publishing in a high-impact journal carries the implicit promise that the article will also be highly cited. However, not all individual papers published in these high-impact journals will fulfil this promise. In our paper Proof over promise: Towards a more inclusive ranking of Dutch academics in Economics & Business, we therefore set out to create a ranking based on “proof”, i.e. rather than looking at the promised number of citations implied by the journal impact factor or article influence score, we look at actual citations to an author’s work.

George-peabody-libraryImage credit: George Peabody Library by Matthew Petroff  (Wikimedia, CC BY-SA 3.0)

Introducing two new impact metrics with individualized career focus

In our paper we use two relatively new metrics, the citations per author per year (CAY) metric and the individual annual h-index (hIa, see Harzing, Alakangas & Adams, 2014 for details). These two metrics are the individualised and annualised equivalents of respectively total citations and the traditional h-index, two important bibliometric measures. We correct our metrics for the number of co-authors as the number of co-authors positively influences both the number of publications an academic is able to publish and the number of citations. A correction for the number of years an academic has been active is important as citations continue to increase over an academic career.

hIa career impact: field independent focus on contribution

In this blog, we focus on the hIa index, which is calculated by dividing the individual h-index (an h-index corrected for the number of co-authors) by the number of years an academic has been active, i.e. the number of years that have lapsed since their first publication. The metric thus represents the average number of single-author-equivalent “impactful” articles that an academic has published per year and hence permits an intuitive interpretation. Based on an empirical example of 146 academics in five major disciplines at different career stages, Harzing, Alakangas & Adams (2014) showed that the hIa index attenuates h-index differences that are purely attributable to (disciplinary) co-authorship practices and career lengths. As such, this metric provides a more reliable comparison between academics in different disciplines and at different career stages than the h-index.

A second aspect of our focus on proof over promise is the use of Google Scholar rather than ISI as a data source. Although ISI listing is seen by many to imply a quality stamp, in our view it should not matter where research is published. If a particular research output is highly cited, it clearly influences the field and that should be more important than the journal in which it is published. As Google Scholar on its own is not very suitable for bibliometric analyses, Publish or Perish (Harzing 2007) was used to collect citation data from Google Scholar. There are now more than 500 published articles referring to the Publish or Perish program. This indicates that—in spite of its limitations—Google Scholar is perceived to be a useful source of bibliometric data.

Resulting ranking for Economics & Business in the Netherlands

The resulting hIa-based ranking of academics in Economics & Business in the Netherlands (see Harzing & Mijnhardt, in press for details) is substantially different from the original ranking based on publications and is more inclusive in terms of disciplines, age and affiliation.

  • Whereas in the original publication-based Economics top-40 more than two thirds of the listed academics works in Economics or Management Science, this proportion is reduced to 43% in the hIa ranking. The change in disciplinary composition is particularly striking in the top-20. In the publication-based Economics top-40, three quarters of the academics in the top-20 were economists, in the PoP hIa top-40 this is reduced to just over a third.
  • At 47, the average age in the hIa top-40 is lower than in the original ranking (50), but most noticeable is the addition an additional five academics under 40 and another five aged between 40 and 45. In fact, all but three of the newly listed academics in this top-40 are aged 45 or under.
  • In the original top-40 academics affiliated with Erasmus and Tilburg made up just over or just under half of the list, in the hIa top-40, this is reduced to a third. Apart from the University of Utrecht, every university is represented in the hIa top-40. Maastricht, Eindhoven and the UvA (University of Amsterdam) in particular do much better in citation-based rankings than in the original publication-based top-40.

We argue that our “proof over promise” approach is more “democratic”/inclusive than the original Economics top-40. First, by expanding the type of research outputs considered beyond the narrow scope of publications in ISI-listed journals, we remove the disciplinary bias against Management, Marketing and Accounting & Finance, disciplines in which a smaller proportion of high-quality journals is ISI-listed than in Economics and Management Science (Harzing & van der Wal, 2009).

Second, citation-based performance metrics can be argued to be more democratic as their “verdict” is based on the reception of the paper by the academic community as a whole, whereas acceptance in a high-impact journal is dependent on only a handful of gatekeepers (the editor and reviewers).

Third, our ranking was conducted with a free software program (Publish or Perish) and a publicly available database (Google Scholar). Hence, any reader can easily replicate the ranking without the need for subscription-based databases. This also means that any academic can look up their own citation record and easily find out where they score in the current ranking

Fourth, citation-based rankings and in particular the hIa ranking are likely to provide more dynamic rankings in terms of changes over the years. Younger academics can more easily enter into the hIa ranking if they perform well relative to their career length in terms of single-author equivalent impactful papers.

As for other research metrics, the CAY or hIa-index should never be used as the sole criterion to evaluate academics. Another crucial question that should always be asked is: “Has the scholar asked an important question and investigated it in such a way that it has the potential to advance societal understanding and well-being?” (see e.g. Adler and Harzing, 2009). However, we argue that the hIa-index provides an important additional perspective over and above a ranking based purely on publications in high impact journals alone.

Implications for Business schools

Using metrics such as the CAY and hIa, business schools can move away from the journal-based logic and start thinking from an impact -driven logic. This could have important consequences for incentive policies, mainly for senior faculty. We advice deans of business schools to start assessing their senior faculty from an impact perspective, instead of the narrow journal-based assessment. This approach is not only more inclusive, it is also more democratic and gives more opportunity for diversity to flourish, especially in business schools with a diverse social sciences background.

Further details: Harzing, A.W.; Mijnhardt, W. Proof over promise: Towards a more inclusive ranking of Dutch academics in Economics & Business, in press for Scientometrics, DOI: 10.1007/s11192-014-1370-z, prepublication version available at http://www.harzing.com/papers.htm#top40

Note: This article gives the views of the author, and not the position of the Impact of Social Science blog, nor of the London School of Economics. Please review our Comments Policy if you have any concerns on posting a comment below.

About the Authors

Anne-Wil Harzing is Research Professor and Research Development Advisor at ESCP Europe and Professor of International Management at Middlesex University. Her research interests include international HRM, HQ-subsidiary relationships, the role of language in international business, and the quality and impact of academic research. Since 1999 she maintains an extensive website (www.harzing.com) with resources for international management and academic publishing.

Wilfred Mijnhardt is Policy Director at Rotterdam School of Management and Executive Director of the Erasmus Research Institute of Management (ERIM) at Erasmus University Rotterdam, one of the prominent research institutes in Research in Management in Europe. He publishes a weblog “Research Management in Management Research” and shares his ideas on social media @wmijnhardt

Print Friendly, PDF & Email

About the author

Blog Admin

Posted In: Citations | Evidence-based research | Impact | Rankings

8 Comments