LSE - Small Logo
LSE - Small Logo

Blog Admin

July 19th, 2017

The 2014 REF results show only a very weak relationship between excellence in research and achieving societal impact

6 comments | 6 shares

Estimated reading time: 5 minutes

Blog Admin

July 19th, 2017

The 2014 REF results show only a very weak relationship between excellence in research and achieving societal impact

6 comments | 6 shares

Estimated reading time: 5 minutes

Results of the 2014 Research Excellence Framework have, in some quarters, been interpreted as evidence of a direct relationship between the quality of scientific outputs and the degree of societal impact generated by researchers. However, such an interpretation, allied to definitions of impact such as that used by Research Councils UK, arguably promotes a stronger reading of the REF results whereby only excellent science leads to high-quality impacts. Richard Woolley and Nicolas Robinson-Garcia looked again at the relationship between the 4* components of evaluated scientific outputs and impacts and found a great deal of variation across all panels. It may be that it is this diversity and heterogeneity that constitutes the strength and vibrancy of UK university research.

Much attention has been paid to the relationship between excellent research and societal impact and how to interpret it. Steven Hill describes how Research Excellence Framework unit submissions that performed well on assessment of scientific outputs also did well on impact [1], whilst noting, correctly, that this does not provide any indication regarding the relationship between the two assessed items. Rather, what we can know is whether (or not) highly regarded science and high impact is being produced by the same unit submission. Recent work by Emma Terämä et al found a positive correlation between impact scores and overall research quality scores in the REF, interpreting this sensibly as a sign that generating societal impact does not substitute for the production of scientific excellence.

One way to read the REF results is thus that there is an apparent relationship between the quality of scientific outputs and the degree of societal impact generated by researchers in the UK. Moderate voices preach restraint in interpreting this relationship. However, Research Councils UK (RCUK) defines economic and societal impact as “the demonstrable contribution that excellent social and economic research makes to society and the economy”. This can promote a stronger reading of the REF results whereby only excellent science leads to high-quality impacts. The overburdened figure of the “excellent scientist” becomes seen as the ideal, a superhuman “engaged in entrepreneurial science, creating excellent knowledge and transferring it into the market through spin-offs, patents, licensing and contract deals”. Such a view, that excellent research and societal impact are inextricably linked, has important policy implications. Indeed, funding criteria now often contribute to a situation in which only those who deliver on both fronts simultaneously, and substantially, can be successful.

Further, the belief that there is a direct relationship between the production of scientific knowledge and the generation of societal impacts can lead to simple “monothematic” criteria to assess the overall performance of the UK university research system and its contributions to society. “Excellent research” is the necessary condition and, perhaps, all we need to worry about.

Yet, do the REF data provide evidence that such a simple relationship exists? We took a look at the relationship between the 4* components of evaluated scientific outputs and impacts for all unit submissions (typically departments), as representing the “excellent” end of the REF spectrum. These are the components of unit submissions that are assessed as “world-leading” (scientific outputs) [2] and “outstanding in terms of reach and significance” (impacts) in their field. For each unit submission the REF provides an assessment of the proportion of its submitted outputs and impact evidence that meets the criteria for 4* standard. We plotted these proportions for all panels and the sub-panels, with the proportion for 4* research on the vertical axis and 4* impact on the horizontal axis.

Figure 1: Relationship between the percentage of submissions meeting criteria for 4* for scientific outputs and impacts, all unit submissions, main REF panels

Figure 1 allows us to view much of the “noise” that tends to be submerged in aggregated correlations between excellence and impact. What is immediately obvious is the wide dispersion evident in all panels. At the extremes, it is evident in all panels that a unit submission can produce high levels of world-leading scientific outputs while not generating any outstanding impact. Equally, a unit submission may generate very high levels of outstanding societal impact while producing little world-leading science or, in some cases, none at all. In between these poles what is noticeable is the heterogeneity within all of the panels. Whilst a positive correlation exists between the percentage of unit submissions’ scientific outputs and impacts that are assessed as 4* in all panels, none of these relationships is particularly strong. Moving down a level, Figure 2 shows the correlation figures for all sub-panels.

Figure 2: Correlation between the percentage of submissions meeting criteria for 4* for scientific outputs and impacts, all unit submissions, by REF sub-panel

There is a great deal of variation evident in the correlation results at the sub-panel level. In Panel A, the immediately striking result is the negative correlation between the scientific outputs and impacts of unit submissions in Clinical Medicine. There is considerable variation in the correlations within Panel B, with the strongest results evident for various engineering sub-panels. Variation is also evident across Panels C and D, with particularly weak correlations evident in the Philosophy and Art and Design sub-panels. By way of further illuminating these results, let’s look at one sub-panel from each of the four main panels (Figure 3).

Figure 3: Relationship between the percentage of submissions meeting criteria for 4* for scientific outputs and impacts, all unit submissions, selected REF sub-panels. Examples discussed are highlighted in red.

It seems apparent that the translational characteristics of research in the Clinical Medicine sub-panel are reflected in the very strong performance of its unit submissions in generating high-quality impacts. Interestingly, two unit submissions had their entire impact submissions assessed as 4*, despite also being amongst those unit submissions with the lowest proportion of scientific outputs rated as 4*. One of these (University of Dundee) submitted 194 journal articles, a patent and six case studies, which provided evidence of varied and impressive impacts, including breakthroughs in cardiac clinic trial procedures and genetic causes of atopic diseases, along with a new approach to drug and chemical safety.

The General Engineering sub-panel has one of the strongest correlations between 4* outputs and impacts, yet a significant number of unit submissions in this sub-panel had their impact case studies assessed as not having any 4* impact. A couple of unit submissions generated substantial 4* impacts without producing any 4* outputs at all. The University of Durham had 70% of its impact submission assessed as 4*, including having made important profit-generating contributions to prosthetic joints and environmentally beneficial contributions to aerospace components, while just 5% of its scientific submission (consisting of 101 journal articles and two patents) was assessed as 4*.

In the Business and Management Studies sub-panel the correlation between unit submissions’ 4* outputs and impacts is around the average for all sub-panels. The highest flying unit submissions in terms of scientific outputs are mid-range on impacts, while the best performers in terms of impacts are middle to upper in terms of scientific outputs. Imperial College London provided one of the best-performing unit submissions in terms of 4* outputs. It submitted 204 publications and had more modest levels of assessed 4* impacts (23%), yet demonstrated important government policy influence and success in diverse contributions to healthcare organisation, telecoms regulation, and company innovation strategies.

In the Arts and Design sub-panel there is a very weak relationship between unit submissions’ 4* outputs and 4* impacts, with many institutions assessed at high levels of one or the other, but not both, and with great heterogeneity in between. The highest rated unit submission (Heriot-Watt University) in terms of impact made innovative and market-driving contributions to products, processes and marketing in the technical textiles sector, while having a relatively small proportion of its scientific output assessed as 4*.

These examples are, of course, just a small taste of the varieties of impacts created in the UK research system from a heterogeneous research base. We would argue strongly that they suggest a different interpretation of the REF results, one that values the great diversity of the UK research system. Creating high-quality societal impacts from UK university research does not only come from producing the highest quality science – this much seems very clear. Submerging this understanding beneath a macro-level interpretation of the REF outcomes in terms of the relationship between unit submissions’ excellent outputs and high-quality impacts is premature, and potentially wrong. Rather, better understanding how the quality of science and the quality of societal impacts are linked is an important topic for future empirical investigation.

Such an understanding also seems vital to nuanced policymaking for one major reason: what if it is precisely the diversity and heterogeneity of the relationship between scientific quality and societal impact at the level of working UK academic groups which constitutes the strength and vibrancy of the UK system? What if policy’s overemphasis on driving excellence leads to a homogenisation of the relationship between knowledge production and the generation of impact? Would this be beneficial, overall, for the quality, efficiency, and effectiveness of UK university research?

Our sense is that it would not. Driving policy in pursuit of an idealised “excellent scientist” (high output-high impact), to whom ever more disproportionate amounts of resources are channeled, would likely reduce heterogeneity in the ways research and impacts are coupled together. Reducing this heterogeneity, which we read as the true significance of the REF results, would likely make it more difficult to address the diverse social needs that depend on the continued myriad, embedded, patchwork of successes of UK university research.

[1] Here we refer to impact as defined in the REF: “change or benefit to the economy, society, culture, public policy or services, health, the environment or quality of life, beyond academia.”
[2] This does not tell us anything about whether a unit submitting to the REF could be considered “world-leading” in terms of its scientific outputs, as distinct from the quality of the specific items evaluated. To know something about a submitting unit’s status as “world-leading” we would also need to know the proportions of 4* outputs being generated by comparable units in universities worldwide. It may indeed be the case that some UK units generate world-leading proportions of 4* outputs and as such could be considered “world-leaders”. It may also be the case that some UoAs proportions of 4* outputs are lower than comparable groups in other countries, in which case they could not be considered world leaders – even though they may conceivably be No. 1 in the UK in their field.

Note: This article gives the views of the authors, and not the position of the LSE Impact Blog, nor of the London School of Economics. Please review our comments policy if you have any concerns on posting a comment below.

About the authors

Richard Woolley and Nicolas Robinson-Garcia are research fellows at Ingenio (CSIC-UPV), a science policy and innovation institute at the Universitat Politècnica de València. This blog is part of ongoing collaborative work on research impact through the Spanish EXTRA project and with the OSIRIS research centre at University of Oslo.

Print Friendly, PDF & Email

About the author

Blog Admin

Posted In: Impact | REF2014

6 Comments