Given the well-known difficulties of measuring the full impact of universities, it may be better to focus instead on knowledge exchange, the process by which the societal impact of scientific knowledge is realised. For Frank Zwetsloot and Anika Duut van Goor, “contract income” – the financial investments made by external parties in contract research, contract education, patents or start-ups in the hope of creating impact – is a meaningful indicator of impact.
Measuring the impact of universities is extremely complicated. The measures need to account for all scientific disciplines and, as well as direct impact, should capture the long-term impact which may occur over a timespan of a hundred years or more. Once you realise it may not be possible to measure the full impact of universities, it makes sense to focus instead on the “road to societal impact” which may be defined as “knowledge exchange”, or “valorisatie” as we call it in the Netherlands; the process by which the societal impact of scientific knowledge is realised.
ScienceWorks started to measure interactions between science and society in 2011 on a bi-annual basis. We began by measuring traditional output indicators in three disciplines:
- The most entrepreneurial university – measuring the size and number of university spin-off companies, jobs created in science parks, and the size of the university’s investment funds. We also included the number of patent applications.
- The most cooperative university – measuring all contract income, license income, public-private consortia, and public-private publications.
- The best communicating university – measuring the appearances of universities in the news on radio, TV and other popular media, excluding scientific literature.
After making a correction for the number of researchers employed by a university, technical universities appeared to score much better in the first two disciplines. The social sciences, however, seemed to be far more popular in the media. Overall, the technical sciences scored better in relation to financial investments and job creation.
For the 2017 ranking, knowledge transfer experts from “classic universities”, meaning universities that teach all scientific disciplines, asked ScienceWorks to devote more time and attention to measuring the societal impact of the social sciences and humanities. Subsequently, we created a new category: “policy impact”. This category measured the references to Dutch universities in the European Parliament, in Dutch national Parliament documents and in municipal council documents, and also the number of advisory functions of scientists in National Government Advisory Boards. Not surprisingly, the social sciences, more so than the humanities, scored higher than the technical sciences and life sciences.
Image credit: FotoshopTofs, via Pixabay (licensed under a CC0 1.0 license).
Balancing the outcomes
As mentioned above, the outcomes of indicators for “entrepreneurship” and “cooperation” are more focused on financial investments, job creation, and the development of start-up companies. The other two indicators are more strongly focused on appearances in the media or in political arenas. We asked ourselves whether these should be equally valued when measuring “overall impact”. It was clear that classic universities would score higher if we did so.
To decide on the weight of each category, we needed to define the societal value of the roads along which scientific insights are disseminated. The publication of scientists’ research outcomes usually falls into one of four different categories:
- Publications in scientific literature.
- Publication in a research or advisory report which is paid for.
- Publication in a popular format, available on the market (book, lecture, article in popular magazine).
- Publication of a new finding in the media.
For the first category, we have no means to measure the impact in society, as it is restricted to the academic arena. For the other three categories we presumed that publications for which the consumer has paid may indicate a higher societal value than a (free) internet publication. That’s why we decided to give the outcome for “entrepreneurship” and “cooperation” – both measuring more business-focused factors – a weight factor of 30%, and the outcome for “communication” and “policy impact” each 20%. Nevertheless, the overall winner of this national impact ranking in the Netherlands – Twente University – would have won even if we had treated these four categories on an equal basis (each weight factor being 25%).
Knowledge transfer versus impact
If we look at our findings, we may conclude that the impact of science can only be professionally measured on a quantitative basis when it is related to a kind of “direct impact”. This includes attention in the media, political arena, or contract-related investments from external parties who would like to get something back for their investment in research or education. It also implies that “contract income”, which in many countries accounts for almost 25% of the funding of university research, is a meaningful indicator for impact. Its income derives from contract parties (business, charities, government-users, European Union) who trust in a concrete kind of “payback”, which will be realised in the near future. This implies that the financial investment in contract research, contract education, patents or start-ups remains the best indicator of societal impact, although there is often no impact yet. It is an investment in knowledge transfer for which the investor itself is responsible for the impact on the end user. This end user is the consumer of a new product, a patient waiting for a new medicine, a policymaker who needs evidence, or a better understanding of society or of ourselves.
The assumption that external investments in university research are “just commercial” and not “societal” is unfounded. The composition of contract research has changed; in the Netherlands, EU investments per researcher went up from 17% to 22% (as a proportion of the total contract research income) and business investments per researcher went down from €9,800 to €7,900 (from 22.5% to 19.5% of the total contract income). All these investors want to see a return on their investments, although the final impact in society is much harder to measure in figures. You may even argue that these governmental and industrial research partners take over the responsibility from universities to generate impact with the research outcomes. We need to professionalise the transfer of science to society and we need to stimulate the quality of these interactions and measure it. That is why we strongly support the Knowledge Exchange Framework that has been developed in the UK. Because we may not be able to measure the final societal impact of science, we should warmly embrace insights on how to ensure the roads towards impact are as concrete as possible.
For more insight on the impact ranking of Dutch universities please read the ScienceWorks statement here.
This blog post is published with the support of the Campaign for Social Science.
UPDATE: this blog post was updated on 20 July 2018 after a section of the authors’ submitted text was omitted in error.
Note: This article gives the views of the authors, and not the position of the LSE Impact Blog, nor of the London School of Economics. Please review our comments policy if you have any concerns on posting a comment below.
About the authors
Frank Zwetsloot is the CEO of ScienceWorks and founder of the Association of European Science- & Technology transfer Professionals (ASTP) and the Network for Advancing & Evaluating the Societal Impact of Science (AESIS).
Anika Duut van Goor is the General Manager of the Network for Advancing & Evaluating the Societal Impact of Science (AESIS).