For many years, academia has relied on citation count as the main way to measure the impact or importance of research, informing metrics such as the Impact Factor and the h-index. But how well do these metrics actually align with researchers’ subjective evaluation of impact and significance? Rachel Borchardt and Matthew R. Hartings report on a study that compares […]
Where are the rising stars of research working? Towards a momentum-based look at research excellence
Traditional university rankings and leaderboards are largely an indicator of past performance of academic staff, some of whom conducted the research for which they are most famous elsewhere. Paul X. McCarthy has analysed bibliometric data to see which research institutions are accelerating fastest in terms of output and impact. The same data also offers a glimpse into the future, […]
The methodology used for the Times Higher Education World University Rankings’ citations metric can distort benchmarking
The Times Higher Education World University Rankings can influence an institution’s reputation and even its future revenues. However, Avtar Natt argues that the methodology used to calculate its citation metrics can have the effect of distorting benchmarking exercises. The fractional counting approach applied to only a select number of papers with high author numbers has led to a situation […]
The effect of NSS scores and league tables on student demand and university application rates is relatively small.
As competition for student recruitment continues to intensify, policymakers and administrators are encouraging an emphasis on ‘student experience’. The National Student Survey (NSS) scores are one indicator that attempts to measure this. But do students really take any notice of NSS scores in making their university choices? Stephen Gibbons shares findings which suggest the effect of changes in NSS scores on […]
There are a surge of rumours circulating over how higher education will be affected by the government’s Comprehensive Spending Review at the end of the November. Responding to the latest suggestions, Martin Eve writes below directly to Jo Johnson, the Minister for Universities and Science. Whilst there are many causes for concern outlined, of particular concern is the emphasis put on teaching at […]
Why did REF2014 cost three times as much as the RAE? Hint: It’s not just because of the added impact element.
The benefits of any research assessment framework should ideally outweigh the costs and burden incurred by universities and staff. Derek Sayer argues there should be cause for concern now that recent analysis shows the 2014 REF bill was three times as much as the last UK assessment exercise. The costly increase in staff time was driven by the increased importance […]
The metrics dilemma: University leadership needs to get smart about their strategic choices over what counts.
The review of metrics enjoins universities not to drift with the ‘metric tide’. To do this requires a united front of strategic leadership across the sector, argues HEFCE’s Steven Hill. Rather than the inevitable claims about league table positions on website front pages, universities could offer further explanation of how the rankings relate to the distinct mission of the institution.
This is part […]
Literacy as Numbers: The efficacy, merits and validity of transnational literacy assessment programmes.
Debates about the nature of literacy and how to account for the diversity of learning are far from resolved. A new book, Literacy as Numbers, looks at how literacy itself is being reframed around globalized assessment regimes. Camilla Addey delves into how these comparable numbers, now so heavily relied on in national policy, are produced, and how they are shaping our understanding of the meanings and […]
Predicting the results of the REF using departmental h-index: A look at biology, chemistry, physics, and sociology.
Can metrics be used instead of peer review for REF-type assessments? With the stakes so high, any replacement would have to be extremely accurate. Olesya Mryglod, Ralph Kenna, Yurij Holovatch and Bertrand Berche looked at two metric candidates, including the departmental h-index, and four subject areas: biology, chemistry, physics and sociology. The correlations are significant, but comparisons with RAE indicate that […]
Proof over promise: Moving citation metric systems beyond journal impact towards a career impact approach.
Publishing in a high-impact journal carries the implicit promise that the article will also be highly cited. But the proof of this logic remains unsubstantiated. By combining more accurate citation metrics, like the hIa-index and the citation-per-author-per-year metric, Anne-Wil Harzing and Wilfred Mijnhardt provide a more substantial alternative to the narrow journal-based metric. This combined metric provides a more reliable comparison between […]
University rankings wield immense influence over Higher Ed and society at large – with positive and perverse effects.
In a time of growing demand for and on higher education, university rankings have transformed university strategy. Ellen Hazelkorn finds their crude simplicity is what makes rankings so infectious. Yet, quality is a complex concept. Most of the indicators used are effectively measures of socio-economic advantage, and privilege the most resource-intensive institutions and-or countries. In response and reaction to the limited […]
Basing university subject rankings on reputation metrics disadvantages departments outside global cities.
University ranking season is always filled with vibrant discussion on the opaque and subjective criteria used to establish the infamous league tables. Jonathan Albright focuses his analysis here on the QS World University Rankings by Subject. By removing the highly subjective score for employer reputation, the results change substantially, especially for universities outside global cities. Each year, parents, prospective students, […]
At yesterday’s Future of Impact conference, Cameron Neylon argued that universities must ask how their research is being re-used, and choose to become the most skilled in using available data to inform strategic decision making. It’s time to put down the Impact voodoo doll and stop using rankings blindly. “Impact” is a word that has gained great power in research policy […]
The demise of the Impact Factor: The strength of the relationship between citation rates and IF is down to levels last seen 40 years ago
Jobs, grants, prestige and career advancement are all partially based on an admittedly flawed concept: the journal Impact Factor. Impact factors have been becoming increasingly meaningless since 1991, writes George Lozano, who finds that the variance of papers’ citation rates around their journals’ IF has been rising steadily. Thomson Reuters assigns most journals a yearly Impact Factor (IF), which is […]
Measuring thoughts and thinkers: why the ongoing conflict about measuring the value of science and humanities may be ultimately fruitless
Small competitions, such as the BBC/AHRC New Generation Thinkers scheme, preserve the essential features of larger assessments of research quality such as the REF, argues Jon Adams. But what does it mean to compare achievements in such disparate fields as history and physics? The BBC and AHRC recently ran a joint competition to locate ten ‘New Generation Thinkers’, a talent […]
A beginner’s guide to the different types of impact: why the traditional ‘bean-counting’ approach is no longer useful in the digital era
Enhancing the capacity to understand academic influence can help all of us in the social sciences to become more effective as researchers. But how do we define one type of impact from another? Here we take a closer look at the differences between academic impact and external impact, a step away from the traditional passive approach to making impact and […]
In March, Brunel University’s Health Economics Research Group (HERG) hosted an international two-day workshop on ‘State of the Art in Assessing Research Impact’. The workshop built on HERG’s leadership in assessing research impact, especially the ‘Payback’ Framework pioneered by Professor Martin Buxton and Professor Stephen Hanney in the 1990s to assess the outcomes of healthcare research. This approach gauges […]
‘Maximizing The Impacts Of Your Research: A Handbook For Social Scientists’ now available to download as a PDF
For the past year a team of academics based at the London School of Economics, the University of Leeds and Imperial College have been working on a ‘Research Impacts’ project aimed at developing precise methods for measuring and evaluating the impact of research in the public sphere. We believe the our data will be of interest to all UK […]
This week THE brings us the results of Australia’s first ever national research assessment, which found research in the sciences stronger than in the humanities and social sciences.
The Excellence in Research for Australia (ERA) ranked biotechnology, earth sciences, physical sciences, chemical sciences, and environmental sciences as the top 5 fields of disciplines; history was the only humanities or social […]