When we think of impact, we must simply think of the ability to ‘re-use’ or recycle our research, writes Cameron Neylon. If we can expand the culture of citation and linking to new objects and new forms of re-use, particularly for objects on the web, then we can gather a much stronger and more comprehensive evidence base to support all sorts of decision making. Start counting those retweets!
I developed an interest in research evaluation as an advocate of an open research process. It was clear that researchers are not going to change themselves, so it seemed that someone was going to have to change them. Funders wield the biggest stick, the only question was how to persuade them to use it.
But of course it’s not that simple. It turns out that funders are highly constrained as well. They can lead from the front but not too far out in front if they want to retain the confidence of their community. And the actual decision making processes remain dominated by senior researchers: successful senior researchers with little interest in rocking the boat too much.
The thing I realized as I dug deeper was that the key to this problem lay in finding motivations that work across the interests of different stakeholders. Could we find objectives that were shared by both funders and researchers? Is there something that unites both researchers and funders, as well as government and the wider community?
Researchers and funders understand impact
I’d like to suggest that one answer is impact. The research community as a whole has a stake in convincing government that research funding is well invested and likewise, government also has a stake in understanding how to maximize the return on its investment. Researchers do want to make a difference to the wider world, even if that difference may be a long way off. We also understand that funding a range of research with no apparent immediate application is critical to making that difference in the long term. We need to support a diverse range of research now in the knowledge that some of it will not produce immediately measurable benefits.
Impact has a bad name among many researchers, but I believe if we can move on from that gut response, then it raises important and challenging questions we need to address. What is research for? What is its role in our society and what is the right level of societal investment? What outcomes would we like to see from it, and over what timeframes? How can we evaluate those outcomes? And against what criteria? The answers include economic impact yes, as well as social, health, policy, and environmental impact. But alongside this list of desirable end points we also need to remember the impact that research has on other research, perhaps its most important near term application.
Thinking of impact as re-use
All these forms of impact have something in common; the idea of ‘Re-use’. What we mean by impact is simply re-use. Re-use in industry, re-use in public health and education, re-use in policy development and enactment, and re-use in research.
And this frame of mind illuminates some interesting possibilities. We can measure some types of re-use. Citation, retweets, re-use of data or materials, or methods or software. We can think about gathering evidence of other types of re-use, and of improving the systems that acknowledge re-use. If we can expand the culture of citation and linking to new objects and new forms of re-use, particularly for objects on the web, where there is some good low hanging fruit, then we can gather a much stronger and more comprehensive evidence base to support all sorts of decision making.
There are, of course, also problems and challenges. The same ones that any social metrics bring; concentration and community effects, the ‘Matthew’ effect of the rich getting richer. We need to understand these feedback effects much better and I am very glad there are significant projects addressing this. Of course these problems also exist for our existing research assessment frameworks and work on new measures will help us to understand what we are already doing.
But there is also something more compelling for me in this view. It allows us to reframe the impact debate around basic research. The argument goes that we need basic research to create future breakthroughs. We know neither what we will need nor where it will come from. But we do know that it’s very hard to predict – that’s why we support curiosity driven research as an important part of the portfolio of projects. Yet the dissemination of this investment in the future is amongst the weakest in our research portfolio. At best a few papers are released then hidden in journals that most of the world has no access to and in many cases without the underlying data. And this lack of effective dissemination is often because the work is perceived as low, or perhaps better, slow impact.
We may not be able to demonstrate or to measure significant re-use of the outputs of the research that we fund now for many years. But what we can do is focus on optimizing the capacity and the potential for future exploitation. Where we can’t demonstrate re-use and impact we should demand that researchers demonstrate that they have optimized their outputs to enable future re-use and future impact. Where research doesn’t have near term applications we should apply a higher standard of curation and archival, and above all require researchers to ensure that their work remains discoverable over the next decades when its possible applications may surface.
Making research outputs truly open
And this brings me full circle. My belief is that the way to ensure the best opportunities for downstream re-use, over all timeframes, is that the research outputs are open, in the Budapest Declaration sense: available for any use by any person with no restrictions beyond a requirement for attribution. But we don’t have to take my word for it, we can gather evidence. Making everything naively open will not always be the best answer, but we need to understand where that boundary is and how best to deal with it. We need to gather evidence of re-use over time to understand how to optimize our outputs to maximize their impact.
The questions around impact, around how to structure our research effort, and how to make decisions on the allocation of scarce resources are not philosophical or theological questions. They are engineering questions. But engineering decisions require a choice to be made about what is to be optimized. I believe that if we choose to value re-use, to value the downstream impact that our research has, or could have, then we can make this debate not about politics or ideology but how about how best to take the public investment in research and to invest it for the outcomes that we need as a society.
This is an edited version of the text that Cameron Neylon spoke from at the Altmetrics Workshop in Koblenz in June. There is also an audio recording of the talk available as well as the submitted abstract for the workshop.