Since the term was popularised by Rittel and Webber in their seminal article, Dilemmas in a general theory of planning, the concept of ‘wicked problems’, or those that are resistant to optimal solutions, has posed a significant challenge to the creation of public policy. In this post, Brian Head, draws on his new book, Wicked Problems in Public Policy, to discuss how approaches to wicked problems and the challenges caused by an increasingly complex and polarised society have developed in recent years and argues for an approach aimed at better managing wicked problems, as opposed to seeking to directly solve them.
Since the 1980s, there has been much support for using better evidence to solve our big policy problems. The idea is simple – evidence helps us to understand the scope of the problem, the causal relationships at work, and the optimal solutions. The evidence-informed approach of an earlier period was summed up in the mantra: ‘what counts is what works’. Subsequently, there have been big investments in the data bases, knowledge networks and analytical tools needed for gathering and analysing information for the policy process. We now have sophisticated skills to track program performance, design policy alternatives, and use ‘big data’ from organizational records and digital platforms to broaden our evidence base for policy-making.
Nevertheless, the optimistic outlook concerning evidence-informed policymaking has been strongly challenged, and a degree of malaise has emerged. There are three key reasons for this shift.
First is the well-known critique of rationalist claims that the policy process should be substantively based on rigorous ‘evidence’. In rebuttal, critics state that stakeholder interests and experience are crucial, that public policy decisions are ultimately political, and that accountability must rest with elected governments. Underpinning this are five common claims:
- evidence is always incomplete, especially for complex and evolving issues;
- evidence includes many forms of knowledge, which are often inconsistent;
- evidence is always subject to value-laden interpretations, hidden or overt;
- evidence is inevitably “cherry-picked” by governments and stakeholders to support their preferred positions; and
- reliance on specialized expertise implies technocratic and undemocratic styles of decision-making.
The second key reason for the malaise is the rapid rise of mass communication channels that propagate misinformation, personal beliefs, and vilification of opponents. Polarisation intensifies in-group bias and provides excuses for ignoring alternative information. Partisan polarisation undermines community confidence in procedural fairness and legitimacy, which are crucial for trust in public institutions. In the face of this tsunami of propaganda and incivility, the enlightenment model of reasoned debate in the public sphere has been deeply wounded. In responding to this wave of ‘post-truth’ anti-science, the defenders of evidence-informed debate and civic education have created new networks to discuss strategy, share information, improve their communication of evidence-informed policy ideas, and develop new tools for fact-checking or reviewing the claims of partisan advocates.
The third key reason for concern about the future of evidence-informed policymaking is that so many policy problems are intractable, controversial and turbulent. This is the domain of ‘wicked’ problems
The third key reason for concern about the future of evidence-informed policymaking is that so many policy problems are intractable, controversial and turbulent. This is the domain of ‘wicked’ problems, as outlined in Wicked Problems in Public Policy. Wicked problems are characterised by complex interactions, gaps in reliable knowledge, and enduring differences in values, interests and perspectives. Unfortunately, ‘more science’ cannot resolve these conflicting views, and therefore more data cannot directly help to de-politicize the partisan divide. Wicked problems include long-standing, yet continually evolving problems such as: refugees and immigration, human rights and inequalities, climate change, food security, water and energy security, biodiversity protection, terrorism, and the peaceful resolution of major disputes.
In my book, I argue that some policy issues are more amenable to evidence-informed solutions than others. In some cases, administrative and technical policy issues can be settled quickly, if, there is an agreed evidence base and minimal conflict about values. However, for big problems marked by complexity, value divergence and knowledge uncertainties, the standard decision-making processes do not lead to effective solutions; hence, increasing demands for new thinking.
What forms of policy innovation are most useful to tackle wicked problems? Because the framing or scoping of these problems is strongly contested, it is especially difficult to find acceptable solutions. Calls for new policy thinking are increasingly common, but there are diverse views about how to generate policy solutions.
Most of the innovation literature highlights processes for engaging stakeholders and experts in various forms of collaborative dialogue. Sharing information and ideas – about values, goals and strategies – is always a sensible response to complexities. This applies not only to the well-established problems (like entrenched inequality and ecological sustainability), but also to newly emerging crises and challenges. For example, how should policymakers respond to the next pandemic; or respond to the regulatory challenges of new industries based on digital data services or therapeutic biotechnologies?
The case for using social experimentation and co-design processes is mainly pitched at the local level, rather designing mainstream national programs. These collaborative agendas and methods are very diverse. Choices need to be ‘fit for purpose’, working with relevant stakeholders and adapting to the nuances of each challenge. Collaborative and coordinated approaches are widely recommended, but the difficulties of successfully implementing such approaches are well known and skilled leadership is needed.
Finally, it is true that there are no neat and correct answers to wicked problems. They might not be solvable in the short term. But there are ways to better understand and manage them.
Politicians often prefer a quick fix or a simple ‘solution’ for managing a complex policy problem. In many cases, government leaders will attempt to impose their own preferred solution, either to appease their own supporter base (‘keeping promises’), or to close down the debate. This is often ineffective and sometimes counter-productive. In the long run, differences in stakeholder values and interests need to be acknowledged, and methods found to accommodate diversity and equity. This is a challenge for political leadership, as much as a challenge for evidence and expertise.
Finally, it is true that there are no neat and correct answers to wicked problems. They might not be solvable in the short term. But there are ways to better understand and manage them. For example, inclusive processes for considering the nature of the problems and possible paths toward improvement are often more beneficial than ideological solutions imposed by government. Good quality knowledge and analysis are always useful, but information alone cannot transform complex problems into simple solutions. Information-based strategies are only one important thread in the policy mix required to enable us to tackle the wicked features of complex and contentious problems.
Each problem has a unique history, even though many problems closely interact with others (such as poverty, housing, health and education). Solutions are provisional and contingent. They are temporary political accommodations, depending not only on best available evidence but also on stakeholder perceptions and the capacity of leaders to negotiate shared goals.
This all points to the need for a realistic standard of success in dealing with wicked problems, especially the most difficult ones like preventing domestic violence or responding to global climate change. To call for the definitive solving of these problems is to set up an impossible standard. Shifting from a solutionist perspective to one in which wicked problems can be much better managed, could ultimately lead to significant improvements in addressing the greatest challenges we currently face.
Interested readers can find Brian’s book, Wicked Problems in Public Policy open access here.
The content generated on this blog is for information purposes only. This Article gives the views and opinions of the authors and does not reflect the views and opinions of the Impact of Social Science blog (the blog), nor of the London School of Economics and Political Science. Please review our comments policy if you have any concerns on posting a comment below.
Image Credit: Jukan Tateisi via Unsplash.