‘Pure’ research has the potential to make a big impact on an audience of scholars, but publishers might hold the key to increasing its wider public impact, argues Martyn Lawrence of Emerald Group Publishing. Based on his presentation at the Investigating Academic Impact Conference, he argues that publishers have a vital part to play in increasing academic impact, and uses his matrix to show that applied research boasts demonstrated impact among a much wider audience, despite making less contribtion to rankings.
What are publishers for? An unusual question, perhaps, but one that is increasingly asked in the digital age. Authors including junior and senior academics, heads of scholarly associations, and deans of research, regularly approach us asking what help can be offered to demonstrate the wider impact of their work. If our experience is anything to go by, the general feeling within academia is this: “of all people, you publishers ought to be helping us out. Isn’t this why you’re here?”
As more and more funding bodies have identified the need for wider research impact (witness the aborted Research Quality Framework in Australia, or the very-much-alive Research Excellence Framework in the UK), Emerald has tried to respond to these requests for help. In doing so, we’ve noticed a parallel with our longer-term goals – for more than 40 years, we’ve tried to help authors bridge the gap between theory and practice, and to publish ‘research you can use’.
To this end, we have a number of ongoing initiatives that assist the demonstration of impact.
- Publishing workshops
We have designed and presented numerous publishing workshops, including emphasis on improving abstracts, keywords and titles. For example, we found that titles with 6-10 words have the highest usage, whilst titles over 25 words long are downloaded the least. In an online world, articles compete for readership and visibility is everything. This doesn’t mean the research within the paper should be any less rigorous, but from the very beginning even the title should be seen to be relevant. Invisible research is, by definition, low-impact.
- Research awards
Each of our journals and book series has an annual best paper award. Recently, we have added social impact, practical impact and long-term impact awards, judged across our portfolio of titles. The aim is to better recognise (and thus help to promote) research with these strengths.
- Structured abstracts and repackaged material
Structured abstracts were introduced in 2005, and now list a specific requirement for authors to draw out the research, practical and social implications of their paper. Far from being onerous to complete, structured abstracts help the articles to compete in an age of electronic ‘skittering’. We also offer repackaged short articles for practitioners or other new audiences – full research papers from Emerald journals rewritten by experts for a new readership. Some examples of these can be viewed here.
Despite many of these more long-term initiatives, we came recently to consider impact in the light of the REF and RQF, and the AACSB 2008 report that called for greater practical research impact. If the debate was all about measuring impact, how could we as publishers help? Conventionally, of course, scholarly impact is assessed through citation – a crucial albeit rather one-dimensional measure. As Professor Søren Holm said in a recent round table: “The use of citation measures is perfectly understandable but also completely crazy. One of the easiest ways to get lots of citations is to be interestingly wrong”.
Impact matrix and the 6 Impact Zones
We encourage authors to measure research impact on a much wider scale, using more than this one variable. So we created a rough impact matrix composed of 6 Impact Zones: Knowledge, Teaching and Learning, Practice, Public Policy, Economy, Society and Environment. Each segment has different indicators of impact, many of which overlap. Knowledge, for example, might be assessed by citation, downloads, article usage, breadth dissemination, quality of the editors, or society links. The reason we also downloads so highly is because in our view, a journal receiving 350,000 article downloads per year is having an impact – even if it may not be highly-cited. And even if, as the criticism runs, these are student downloads (not that there is much evidence for this), surely that would score highly in the Teaching and Learning segment? It should also be noted that Emerald articles are counter-compliant, meaning, in short, the downloads are measured to an industry standard and can not be falsified.
Knowledge – scholarship which contributes to the body of knowledge and generates further research. This is assessed by citations, usage, peer recognition and self-stated research conclusions.
Teaching and learning – students and faculty are direct consumers of research. This is assessed by clarity of conclusions to aid learning, provision of case studies and teaching examples, usage statistics and course adoption/curricula change.
Practice – business leaders, practitioners and consultants are all affected by the outcomes of research. This is assessed by university-business collaboration, consultancy application and implications for practice self-stated.
Public policy – state officials, politicians, decision makers in public bodies, institutions and charities draw on research to shape their policies. This is assessed by self-stated potential implications and subsequent policy revisions.
Society and environment – influencing CSR in industry, business and public policy and the incorporation of social/environmental values in research outputs. This is assessed by informing social policy, industry adoption and self-stated implications for society.
Economy – research which contributes to organization-level or macro-level wealth creation and business advancement. This is assessed by: future economic savings, revenue increase and self-assessed business/economic impact.
If you take Knowledge, scored it 1-5 on citations and usage, and balanced the quantitative with an element of peer recognition (and then did the same for other segments), could you come up with a rudimentary score for impact? In other words, a theoretical paper, designed specifically for use by the authors’ colleagues at universities, might score:
Knowledge: 5, Practice: 1, Teaching & Learning: 3, Policy: 2, Society & Environment: 2, Economic 2. Total =16
By comparison, a piece of applied/action research could score:
Knowledge: 2, Practice: 5, Teaching & Learning: 3, Policy: 4, Society & Environment: 4, Economic: 3. Total =21
What this shows, if anything, is that ‘pure’ research has impact, but it is usually limited to a scholarly audience. Applied research makes less contribution to rankings (and perhaps by extension, to tenure or funding) but does have wider impact.
This isn’t a perfect measure (witness objections to the similar Leopold Matrix, for example). There is also the issue of the disruptive paper, the piece of research that challenges current thinking, goes unnoticed for years, and then changes the world. This is usually cited as the crucial objection to ‘impact’ but these papers are rare. So while it doesn’t seem unreasonable for a quantitative measure to take note of the disruptive paper, it does seem reasonable to say that principles shouldn’t be based on exceptions.
Room for improvement
Emerald is delighted to offer this matrix, or an adapted version, to anyone who can use it. We acknowledge that the measurements will almost certainly need to differ from subject to subject, and in certain disciplines (hard sciences, for instance) it may be inappropriate. It is primarily designed as a piece for discussion: if this doesn’t work, what does? If citation is scored too low, what do we do to remodel the matrix?
Publishers have a role to play in demonstrating and measuring impact, and Emerald is exploring ways in which we can respond to the current debate. Can we solve the challenges faced in the lead-up to the REF? No – in the sense that Emerald is an international publisher, and can not design a model for the UK alone. Our authors and readers are based worldwide, and we pride ourselves on global dissemination. But yes, we can in the sense that as an international publisher involved with its subject communities we should be able to contribute to the most important assessment in the calendar. If your published article won a social impact award or your published article was adapted for use on Emerald’s First Management site, that is impact, and it should be acknowledged in the REF. Assisting in this way is what publishers should be all about.
Listen to Martyn Lawrence’s presentation at the 2011 Investigating Academic Impact Conference here or view his presentation here. More podcasts, presentations, photos and twitter results from the Investigating Academic Impact Conference can be found here.