Traditional university rankings and leaderboards are largely an indicator of past performance of academic staff, some of whom conducted the research for which they are most famous elsewhere. Paul X. McCarthy has analysed bibliometric data to see which research institutions are accelerating fastest in terms of output and impact. The same data also offers a glimpse into the future, helping to identify those academics who might be the research stars of tomorrow.
We currently view success and performance in research and higher education through a rearview mirror. And like any view from a rearview mirror, objects may appear closer and larger than they actually are. Institutions and scholars with fame and reputations from decades ago often continue to dominate top rankings, reasserting that which we already know: the status quo.
Reading many university rankings can be a bit like stargazing. Due to the delay in information reaching Earth we look up at stars that are past their former glory or may not even still be burning. And at the same time, we miss emerging and young stars that are already or will soon be in their prime. Most university rankings are more accurate representations of the past than recent or current performance as they are heavily weighted to include things like reputation surveys that really reflect the impact of work done many years ago, or Nobel prize and Fields Medal counts where the time between the breakthrough work being done and the award is often more than 20 years.
Sir Andre Geim (pictured) won the 2010 Nobel Prize in Physics for his work on graphene, just nine years after being hired as a scientist at the University of Manchester. Image credit: Per Henning by NTNU Faculty of Natural Sciences. This work is licensed under a CC BY 2.0 license.
What if there was another way? All institutions and scholars are in motion. They’re all moving forward, but some faster than others. How would things look if you framed the question not in terms of who has published the most or who has had the most impact in the recent past but rather who is accelerating fastest and thus more likely to top tomorrow’s leaderboards?
Drawing upon a growing collection of emerging, increasingly comprehensive online data sources such as Google Scholar, Microsoft Academic, and League of Scholars, this post shows how the use of new data and metrics can shed light on the up-to-date performance of researchers and institutions. There are also clues as to how we may even be able to glimpse into those researchers and institutions with the greatest future potential.
Introduced in 2004, Google Scholar is a comprehensive online global index of research and scholarship. A longitudinal, cross-disciplinary comparison study published last year showed Google Scholar to be a more comprehensive source of publication and citation data across all disciplines than both Clarivate Analytics’ Web of Science and Elsevier’s Scopus databases —the two traditional sources used in all three major university rankings (Academic Ranking of World Universities, Times Higher Education World University Rankings, and QS World University Rankings).
Figure 1: Web of Science, Scopus, and Google Scholar citation coverage by domain. Data source: Harzing, AW. & Alakangas, S. (2016) “Google Scholar, Scopus and the Web of Science: a longitudinal and cross-disciplinary comparison“, Scientometrics.
Microsoft Academic was established in 2006 as a competitor to Google Scholar and offers a similar open online index of the world’s scholarly work. Microsoft Academic has been shown to be more comprehensive than traditional bibliometric indices and has recently undergone a huge revitalisation. League of Scholars is a new online service building on Google Scholar and other sources to create an up-to-date index of the top one million researchers worldwide, with each ranked on both recent performance and future potential across 15,000 fields and topics.
Which universities have the greatest momentum? A new approach to looking at institutional research performance
Consider how investors look at the property market. In most markets, in most years, property appreciates everywhere but some suburbs grow faster than others. And it’s the change in growth or acceleration that many clued-up investors look for. Popular seaside and riverside locations, villages within commutable distance to thriving cities, and inner-city suburbs undergoing urban renewal often command massive growth premiums compared to the average suburb.
Academia, like the property market, also has its share of superstars. A small number of outstanding academics – Nobel laureates, for instance – are akin to properties with river frontage or beach views, and are naturally not always representative of their whole institution. For this reason, growth in median values (or those of the middle-ranked houses) is often quoted as a better guide than the averages as these tend to be inflated by the presence of a handful of superstars.
Figure 2: Russell Group momentum 2017 – annual growth in median author output and impact (size median author h-index). Data sourced from Google Scholar/League of Scholars.
Using League of Scholars analysis based on Google Scholar data, Figure 2 shows the growth of impact and output of the median authors of the 24 Russell Group universities, compared with the median for authors at all UK universities. Here you can see the output and impact of all Russell Group universities is growing handsomely but some – notably University of Exeter, University of Oxford, Imperial College London, and UCL – are moving ahead faster than their peers in terms of impact, and Queen’s University Belfast, University of Southampton, University of Liverpool, and University of Leeds are outperforming the others on output.
This approach is also validated when compared with the UK’s research evaluation exercises. Looking at the 2008 Research Assessment Exercise results and those of the more recent Research Excellence Framework (the 2014 national research evaluation exercise), each of the Russell Group universities that appears in the momentum index’s top output/impact quadrant either improved or maintained its ranking, with Exeter, King’s College London, Southampton and Queen’s University Belfast among the biggest winners.
Figure 3: Russell Group universities’ research evaluation exercise rankings 2008-2014.
These positions also align with other data, including the League of Scholars index of the location of many of tomorrow’s research stars. Using data analytics and its own Whole-of-Web (or “WoW”) index — a ranking of scholars in 15,000+ research topics, with the aim to predict their potential future research impact in those areas — it aims not only to discover who today’s research leaders are but also to identify early-career researchers with high potential for the future, many of whom will become tomorrow’s research leaders. University of Exeter, in particular, has a much higher ratio of “future” stars than the UK university average.
Figure 4: Where do tomorrow’s research stars call home? Percentage of high-potential authors (WoW-ranked). Data sourced from Google Scholar/League of Scholars.
Leading bibliometric scholar and Publish or Perish co-founder Professor Anne-Wil Harzing has outlined how metrics could be used to replace the very manual, long-winded, and costly national research review processes (e.g. the REF) with similar and often better results. In two hours using the free-to-use Publish or Perish software to source citation data from Microsoft Academic, Harzing was able to create a citation-based ranking of UK universities that correlates at 97% with the REF 2014 power rankings.
Many others, including the Chairs of each of the REF 2014 sub-panels, have made stern public warnings about the limitations of metrics and argue that peer review, despite its challenges and cost, remains the best way to assess research quality. And for some areas where current metrics are sparse and weaker — especially in arts, humanities, and some social sciences — this may yet be the case for some time to come.
However, as more and more detailed, timely, and global data is available from online sources including Google Scholar, Microsoft Academic, and League of Scholars, research leaders will be able to gain perspectives and insights into the increasingly fast-moving, mobile world of research and research talent in ways hitherto impossible.
Note: This article gives the views of the author, and not the position of the LSE Impact Blog, nor of the London School of Economics. Please review our comments policy if you have any concerns on posting a comment below.
About the author
Paul X. McCarthy is an Adjunct Professor of Computer Science at UNSW Sydney and co-founder of League of Scholars. His book Online Gravity (Simon & Schuster) explains how the web has changed the fundamental laws of economics and business.