FacebookBufferPocketShare

Think-tanks are in crisis yet there is still a need for evidence-based policy prescriptions. Dr Andy Williamson believes that if think-tanks are to have a greater impact, they must embrace principles of quality, transparency and balance.

I wrote in The Guardian about a perceived crisis in political think-tanks, largely brought on by two factors; the inevitable tidal drift of ideology and the current economic conditions. In this post I’m going to expand a little on the argument I put forward, drawing out what I think are three critical underlying issues; quality, independence and transparency. This commentary isn’t based on any objection to think-tanks (that would be rather hypocritical), rather, I believe an effectively functioning independent think-tank space is a vital pre-requisite to strong democracy. So, at the end, I’ll hint at some ideas for what might be done to solve the problem.

First I’d like to frame the debate by drawing a clear line between academic research and think-tanks. The latter does not simply replicate the former but that does not mean think-tanks should lack standards and demonstrable rigour. Far too much academic research acts as an analysis of record. This is useful but we need good quality applied research focussed on developing strong recommendations for evidence-based action. One is not better (or worse) than the other.

Quality
In academia considerable amounts of low quality research and subsequent publications are produced simply because one has too; publish or perish, as they say. Don’t for one minute believe that peer-review systems protect us from this, they don’t. Nor is academic research an open or level playing field. Journals are largely closed shops, tightly controlled, inaccessible to most.

So, one would hope that an environment based on quality of thought not quantity of output would be different. Unfortunately, this is not the case. The nature of many current think-tanks means that work agendas are driven by funding rather than the need (or desire) for good quality research. Funding also restricts the quality of staff available. Critical thinking is becoming critically endangered.

With a background in commercial consultancy, I know how the ‘big firm’ model works; send in the partners to pitch then, on day one, a two-days-in-the-job graduate walks in the door with a manual under their arm. Are think-tanks any different? In a word… No. They are over-reliant on low-cost junior staff to do a lot of the heavy lifting. This means either junior researchers or, more often than not, interns. Think-tanks are staffed by a sea of young, eager researchers all keen to make careers in government and politics.

They are undoubtedly smart – the system is so competitive that even for short internships you have the pick of the crop. However, these junior researchers lack the most important element of critical thinking: experience. A lot of theory but zero experience in how to apply it (here’s a perfect example of the theory/practice gap at work). They have no idea how to translate thought into action because they have never worked in or been a stakeholder in any of the systems or policy areas they are working on. They also tend to have limited understanding of research methods and therefore underestimate the importance of good research design.

Of course, the reality is not quite as bad as I make it sound; senior staff oversee this work (don’t they?). But to recognise that the intellectual power-base of many a think-tank is in fact this year’s harvest of new grads on three month rotations should make anyone question the applicability of their findings a little more closely.

Independence
How independent is the research you are reading? I don’t just mean the blatant ideological pap of closet-lobbyists that we regularly see in the US and, increasingly, are appearing in the UK. This stuff is easy to dismiss, and should be!

Don’t confuse independence with objectivity. A level of subjectivity exists in social research, indeed I’ve written quite extensively in the past about the impossibility of undertaking applied research without acknowledging some level of personal and subjective bias. Nor is this a major issue so long as one is clear about the bias and disclose it upfront. And, most importantly, the researcher must remain receptive to data and ideas that will change their opinion.

When all you are doing is producing ‘research’ to back up an unchangeable ideological belief it is not worth the paper it is written on.

But this is an extreme. There are longer shadows at play, more difficult to spot. Some are more problematic than others but all need to be acknowledged if research is to be credible. Is there a familiar party or ideological bias? Is it enough to prevent coming to the ‘wrong conclusion’? Some bias can be dealt with if it is declared and if the think-tank is rigorously managing its processes and its own independence. Being open and transparent lets the reader decide too.

More insidious, more dangerous because it’s about direct funding (the latter point is ultimately about indirect funding), research funded by government departments, through commercial sponsorship, donations or from trusts presents a danger. How much does the need to maintain a funding stream impact on one’s ability to be totally honest in research? I can certainly say I have felt pressured to dilute findings that might be seen as critical of a funder. Something I refuse to do if the data supports the argument I’m making but this position needs to be made clear to the funder in advance.

Transparency
Running alongside quality and independence is transparency. This comes in to play at a number of levels for the think-tank but, in a broad sense, it’s the outside world’s way of establishing the veracity of the work. It helps us to understand the contribution and usefulness of the work done and also the position taken in terms of independence and bias.

It is important to be upfront and honest about why research is being undertaken; who commissioned it and why. It is equally important to be clear and open about how data has been collected, not just from where (and who) but how the data was derived. Issues of method and analysis are important to us understanding what research is trying to say.

Publicly funded academic research usually requires the datasets to be published in an online repository. How many think-tanks do this too, even when their research has been publicly funded? Some do, but more should consider it. It might be as simple as publishing raw survey data in Excel or SPSS file formats for other researchers to use. This can also be useful for checking the veracity of the findings – this is not something to be concerned about if you have followed good principles; just because I’ve re-analysed your data and come to a different conclusion it doesn’t mean your own analysis is wrong, it just means I’ve interpreted it differently.

Surely this is a good thing as it adds to the intellectual debate?

Remember, research is subjective. This is where we see the researcher’s bias and the importance of disclosing method and philosophical or analytical lens becomes important; show me your data, tell me how you approached it, justify your conclusions. I don’t have to agree with them, although I might. What think-tank’s do should contribute to a wider intellectual policy debate beyond narrow ideology and to do this the source of the work must not be obfuscated.

An alternative model?
If these are the fundamental problems that prevent think-tanks being as effective as they could be, what is the answer? I want to conclude with some thoughts on alternatives that might allow some of these problems to be overcome.

Of course, the biggest problem of all is monetising critical thought (for academics as well as think-tanks). The two big expenses are premises and staff but I would rather be impressed by the intellectual capacity of the product than the architecture of the offices. Staff costs explain the drive towards low or no-cost staff. But this affects quality. So I think the answer here is somewhat self-explanatory; start moving towards virtual think tank models. Dump the Georgian architecture and draw in the best thinkers to solve the problems at hand. Obviously that’s a simplistic statement and needs more thought, but it works in other disciplines (such as management consultancy).

We can draw on some principles behind social networks, crowd sourcing and gaming theory to manage the people, process and to produce intellectually rigorous work.

There are well developed models of open publishing that will allow not just finished work but also the underlying thought processes and data to be published so that they are transparent and accessible to anyone. Transparency can go beyond data to include analysis too.

Balance is often missing. This can be restored through the virtual, socially networked model I’m hinting at by bringing together different ideological biases to tackle the same problems. There are ways to manage this process virtually that can assure the outcome is rigorous and free of overall bias or narrowed thought. Veracity and credibility can be assured by unpacking, drilling-down through what has been written to understand on what the assertion is based – opinion, meta-data, fact.

Surely anyone wanting to really understand all sides of a policy issue would value a product like that and with lower overheads it would also be a more commercially viable proposition too?

Print Friendly