Many possible Impact Cases are likely to raise issues about whether they are sufficiently grounded on what the REF audit Panels can recognize as ‘2* research’. The Funding Council’s overly restrictive ‘physical science’ view of how research influences policy has created an artificial minefield of pointless obstacles. However, Jane Tinkler and Patrick Dunleavy show how with skill and persistence, departments can navigate successfully past these dangers.

From 2014 the Higher Education Funding Council forEngland(Hefce) will give a fifth of all future QR monies to universities depending on how far departments have achieved impacts outside academia, assessed via case studies. From the Funding Council’s point of view it is vital that they demonstrate to the Treasury a tight causal link between

• departments or academics getting QR funds at time T1,
• then undertaking research at time T2, and
• subsequently achieving impacts at Time T3.

In the official discourse-world: prior funding creates (at least indirectly funded) research; and funded research in turn then creates external impacts.

Put another way, all the external academic impacts claimed in case studies must be shown to be underpinned by research, on which Hefce guidance says:

‘ “Underpinned by” means that the research made a distinct and material contribution to the impact taking place, such that the impact would not have occurred or would have been significantly reduced without the contribution of that research’ (2011: para 161c).

In some ways this looks like it should be the easy bit. What research did you do that led to the interaction with external bodies that during 2008 to 2013 had effects outside academia by influencing something?

But when is research really ‘research’?

However, it is not quite so simple. For many social scientists (perhaps most?) interactions with government or business are actually based on their whole body of work; or their cumulative experience and expertise on a particular subject; or a combination of both. This expertise may not link back exactly (or in a specifically traceable way) to a given research book or article to be referenced in section 3 of Hefce’s impact form.

For some people, their research is still on-going and a published output is not yet available. For other academics in fact holding advisory positions is (or was) key. For still others, undertaking consultancy or maintaining a detailed partnering kind of involvement with a government agency, firm or NGO is part and parcel of the research accumulation process. It is an essential stage in research-collection and data-acquisition, a critical part of how academics learn what is going on in their area of society, government or the economy, and how its key organizations envisage and address issues or solve problems.

But in Hefce terms, this is a research excellence framework, and so the impact must be shown to be grounded on to between one and six specific pieces of academic work or new knowledge discovery. Departments and researchers have to choose outputs that are directly relevant to the impact being claimed, and they must be able to show with evidence why that is the case.

The 2* quality standard

To know what research you would like to include as ‘underpinning’, you also need to think about the evidence-standard or quality of these outputs. As Patrick Dunleavy has already discussed in this series, Hefce’s discourse has to make continuous over-claims about the ‘quality’ of the work being funded with government support, embodied in its notorious scale running from 1* (work of ‘national significance’ only) through to 4* (apparently world-beating).

Officials do at least recognize that most ‘underpinning research’ is going to be a lot more applied, and perhaps much less formally published or disseminated, than mainstream research outputs. So they have accordingly set a lower threshold for impacts-related research. It must be of at least 2* quality, which means ‘quality that is recognised internationally in terms of originality, significance and rigour’. This is a flat threshold – underpinning research only needs to reach and pass. There is no need to go further and demonstrate any higher quality level.

Hefce also recognizes that the time lags for research to influence outside activity are often long. So the REF rules allow ‘underpinning’ research to cover not just the REF impacts window itself (which is 2008 to 2013), but also all the way back to 1993.

Demonstrating 2* quality is especially straightforward if your ‘underpinning research’ outputs date from 1993 to 2007, were included in previous Research Assessment Exercises (which covered this whole admissible period), and were rated by panels at the time as 2* or above. Other post-2007 academic publications in books and journals that have been peer-reviewed or assessed in some rigorous way may also easily meet the 2* criterion

But what if none of this is the case? There are very good grounds for believing that in social science most of the underpinning research both may not fit the orthodox ‘black literature’ model and may never have been assessed in the RAE before. Key categories here include the following:

• Lesser items in researchers’ output profiles. Many prolific academics doing basic or pure research also undertake consultancy or applied research extensively. But successful researchers (with more ‘pure research’ items to submit) are unlikely to choose applied research items as part of their four main assessed outputs, both for the earlier RAEs and now for the REF. Hence these applied publications will rarely have been externally graded against Hefce’s criteria.
•  ‘Grey literature’ reports, working papers, conference papers, specialist blogs and so on have not been formally published by a journal or book publisher. Instead they are issued as reports, working papers or briefing documents by university departments, professional bodies, private sector firms, government agencies, international bodies, charities, NGOs and so on. In the past, such items were relatively rarely sent to RAE panels (except in engineering and IT studies), but they are overwhelmingly dominant in the field of applied social science.
• The older such unassessed items are the more the problems multiply. If your underpinning research dates back to 1999 or even 2005 (say), how can anyone now say with any certainty that back in the day this was of 2* quality or above? If older unassessed material still reads as excellent and well above the threshold, perhaps there is no problem. But what if it reads as now rather commonplace or superceded, because in the interim research in the field has changed rapidly or moved on?
• Because 2* means ‘internationally recognized’, the problem is even greater.  If departments send any older items out to external referees, they must ask them to do an heroic mental reconstruction of what theUK standards were way back then, at the date of publication. Asking someone overseas to do that now would be exceptionally bizarre and difficult to explain. In our view is just not practical to undertake.

Critically assessing your own underpinning research

It seems inevitable that for hundreds (perhaps thousands) of previously unassessed items of underpinning research, universities and departments will have to undertake their own quality checks. Our advice would be as follows:

• For grey literature and research undertaken on any kind of  paid or consultancy basis (much of which is entirely excellent but not written within the opaque conventions of academic life), departments must carefully assure themselves that the research methods used were sound for the times, and that the arguments made are logically set out, and presented in an evidence-based way.
• Some government bodies, international organizations, professional committees, and also elite businesses, will also get applied research or consultancy work that’s been done for them assessed by their own quality assurance processes. Often this will include detailed in-house and external refereeing, and for government or official audit reports often very rigorous ‘clearance’ and fact-checking processes. If any such processes applied to their underpinning research, academics and departments should always draw the Panel’s attention to it.
• Similarly research may have been subject to external review as part of a research project or programme. Work undertaken for Research Council programmes or connected streams of research run by foundation funders has often been subject to external scrutiny or grading that you can quote in support of it being of 2* quality.
• Note also that both Panels C and D specify that ‘evidence that an output has been highly cited and has formed a reference point for further research beyond the original institution’ would also be an acceptable indicator of 2* quality or above. You can immediately assess how highly cited all your work is by joining Google Scholar Citations (which also now has excellent Updates that prompt you on other research close to your own). Or Harzing’s invaluable software (‘Public or Perish’) also gives a full report on your citations.
•  If none of the above apply (as will frequently be the case), the only solution for validating 2* quality is to conscientiously conduct your own reviews at department, faculty or even university level. Every single publication or output referenced in section 2 of the Impact Case form (including formally published but unassessed items) should be checked by two reviewers – one from within your department, and the other from a different university entirely. (But stay within the UK – going overseas would be silly now for reasons set out above.)
• Your reviewers will need to be carefully briefed about what a 2* standard means in Hefce speak, set against the original date when an output was published and the prevailing literature standards then. Reviewers should be asked to ascertain ‘2* at the time’ quality only by establishing that the assessed work is broadly comparable with other applied work of that vintage which is known to be 2*.

Assuming that underpinning work passes these hurdles, it is completely pointless to obsess about or pay too much attention to the part of the 2* rubric that says the work is at least of ‘international’ significance. Applied social science work is chiefly about the ‘here and now’ and usually relates to a specific immediate, local, regional or national problem. It is definitely not about reaching out to foreign shores and distant academic audiences. So if your underpinning research stands the test of good external refereeing, then just be confident to declare its 2* quality – while clearly showing the HEFCE panel that you have conscientiously checked this aspect and been satisfied.

Writing about your underlying research

Section 2 of the Impact Case Study form allows you only to give a short (maximum500 words) overview of the ‘academic’ part of your impact narrative that includes only the following elements. (You should save all the detailed references and quality-demonstrating sections for section 3 of the form):

What research was undertaken? Briefly outline what the research was, and how it was conducted. It could be a large-scale programme of work that was worked on by single or multiple research teams. Or it could be a body of work from an individual academic. Applied research is often undertaken jointly with external bodies, and much consultancy work by academics is really just applied research focused on a single organization’s problems. If you were generating new knowledge using systematic methods, then that is research.

When was research undertaken? It must lie within the 1993 to 2013 period. If some of the work pre-dates 1993, it is too old to count now, even if some part was undertaken in the mid 1990s. If it is very recent, how can the linkage from the research to the impacts have already taken place? With very recent research, are you not forecasting impacts yet to come, more than demonstrating that they have already happened?

Which staff were involved in the research, exactly? All kinds of research staff can be included in impact case studies, and not just the normally REF-able lecturers and tenured academic staff. So if a research team includes research assistants, post docs, visiting academics etc, they can all be included. Especially for more time-distant research, it is vital to be clear when people joined or left the department and university, and joined or left the relevant research team (two different matters). If research was undertaken jointly with departments within a university that fall in different ‘Unites of Assessment’, or between universities, or with staff or facilities in external partners (like firms, government agencies or NGOs), this needs to be clearly outlined.

Who can claim credit for the research? The Hefce guidance asks for a self-assessment to be made of what contribution each group made to the research. This is often tricky to do, and agreeing it across research teams will no doubt need a bit of diplomacy. But it is as important not to over-claim as to under-claim. Social scientists still co-author much less than physical scientists, but most applied, large-scale and forefront work is now done in multi-actor teams, and will play into complex multi-causal environments. Hopefully, the official Panels will be aware of that.

What were the key (‘bottom line’ or ‘take away’) findings of your research? And why should we care? Don’t go into specifics such as individual research conclusions (which should be includes in section 3 of the form) and be careful about writing up self-aggrandising assessments about the quality of these pieces. Instead, in an accessible way describe what you found and why it mattered to the external audience that your impact claim involves.

What is the wider context in which your research sits? You should also give here any relevant contextual information about your area of research. For all applied work it is probably wise not to assume that each non-academic member of the assessment panel will know in detail about your sub-field of the discipline. Remember that it is the non-academic panellists who are leading on the impact section of the assessment. So make sure you give them a clear sense of why this area of research is significant and how exactly your work or research stream fits into that.

Providing references to your research

In Section 3 of the Hefce form, you need to provide up to six references to published outputs from your research. Select those 2* outputs that are most closely connected to the impact you are about to describe. Remember these might not be a researcher’s most highly cited academic work, or pieces for which she is most well-known. Are you sure the Panel can access these pieces ? If not, make sure a version of the outputs is available should the Panel ask.

The REF Advice Note Series covers all stages of writing, editing, evaluating and upgrading Impact Case Studies.

1: Understanding Hefce’s definition of Impact

2: Identifying ‘possibles’ for your Impact Case Study

Note: This article gives the views of the author(s), and not the position of the Impact of Social Sciences blog, nor of the London School of Economics.

About the author:
Patrick Dunleavy is Professor of Political Science and Public Policy at the London School of Economics and Political Science, where he has worked since 1979. He has authored and edited numerous books on political science theory, British politics and urban politics, as well as more than 50 articles in professional journals.

Jane Tinkler has been PPG Manager since September 2005. She is also actively involved in research projects undertaken by PPG. Previous to this, she was Managing Editor of the journals, Political Studies and Political Studies Review for six years. She also was a Research Fellow in the School of Public Policy, University College London.