Evaluation researchers are tasked with providing the evidence to guide programme building and to assess its outcomes. They labour under the highest expectations – bringing independence and objectivity to policy making. Have these efforts produced a progressive, cumulative and authoritative body of knowledge that we might think of as evaluation science? This is the question addressed by Ray Pawson. Lawrence Buhagiar finds that this book certainly deserves to be widely read and discussed.

The Science of Evaluation: A Realist Manifesto. Ray Pawson. SAGE. February 2013.

Find this book: 

The Science of Evaluation: A Realist Manifesto is an effort to render evaluation research more pertinent than impertinent. To accomplish this goal, Ray Pawson, a Professor of Social Research Methodology in the School of Sociology and Social Policy at the University of Leeds, argues for what some may deem a heretical transformation of evaluation research. Not simply an elaboration and refinement of the ideas in Realistic Evaluation (with Nick Tilley) and Evidence-Based Policy: A Realist Perspective, The Science of Evaluation constitutes both a litmus test for those who undertake evaluation research and a shibboleth for evaluation as a quasi-scientific discipline and its disciples.

The objective of evaluation research, Pawson maintains, should be to understand “why” and “when” interventions work; or put another way – what works, for whom, and in what circumstances. Instead of a dogmatic adherence to an experimental design – as if what is being evaluated occurs in a vacuum – evaluation research should strive for the widest possible understanding by drawing upon an array of potential influences that shape the outcomes of policy and programme interventions.

In Part I, ‘Precursors and Principles’, Pawson outlines the underlying assumptions and principles of his realist perspective. Pawson brackets his version of realism by way of Karl Popper’s “we do not know, we can only guess” and Donald Campbell’s “organized distrust produced trustworthy reports”. For Pawson, a realist approach goes beyond simple identification of exceptional events or patterns and encompasses an iteratively building and testing of theories capable of explaining these events and patterns. By starting with the assumption that there is no such thing as a perfect inquiry, Pawson shrewdly creates the necessary space and reason for his realist manifesto.

Through a survey of classic and modern texts that highlight realist perspectives within the very scientific roots of evaluation, The Science of Evaluation seeks to liberate evaluation research from a disciplinary straitjacket, tailored too narrowly from the stiff fabric of experimental research design. Leaning heavily (perhaps too heavily) upon Popper’s admonition that “science is informed guess work”, Pawson argues from the middle against both absolute and relative truths. All knowledge is partial but can accrue over time. Truth is an accretion, which can be improved upon, but can never achieve the status of a definitive capital “T” truth.

In Part II, “The Challenge of Complexity” is used to build the case for his realist evaluation manifesto. Pawson finds the four main approaches used to deal with complexity to be wanting: augmented trials underestimate complexity; a systems perspective entails a level of abstraction that is too remote; a critical realist perspective uses philosophical smoke and mirrors to achieve ideological ends; and a pragmatic perspective fails because it can only ever offer a plural, piecemeal and contested response to complexity rather than an actual solution.

Prior to commencing an evaluation, Pawson argues that the contours of complexity, as they relate to the intervention(s) under investigation, need to be mapped. If evaluation research is to lay claim to being a science, it must accept complexity as a normative feature rather than a confrontational threat. Instead of attempting to deal with complexity through experimental research design, which only further removes the ‘air of reality’ from evaluation, the discipline should embrace the Popperian notion of science as “informed guesswork.

In Part III, “Towards Evaluation Science,” Pawson proposes his realist perspective as a solution to the challenge of complexity. This is where the ‘how’ of realist perspective is revealed. In Chapter Eight, “Synthesis as Science: The Bumpy Road to Legislative Change,” drawing upon recent research on the effectiveness of laws banning smoking in cars carrying children, conducted in collaboration with the UK National Institute of Health and Clinical Excellence (NICE) and funded by the Economic and Social Research Council (ESRC), Pawson provides a “show & tell” explication of how a realist perspective informs method. Central to the realist perspective is the reconstruction of programme theories that underpin specific interventions.

A realist perspective of evaluation science maintains that our knowledge can never be absolute or complete; at best, it is always partial. Science provides conditional truths rather than irrefutable Truth. Policy-making is a recursive, discontinuous process involving many different steps, and a myriad of factors over a considerable length of time. It is never as simple following the evidence. Evidence has to be interpreted and adapted to the local circumstances. Even the will to make changes towards a better world do not always have obvious choices.

Despite its thoroughness, The Science of Evaluation is not without some weaknesses and limitations. For example, Pawson focuses exclusively on mid-range theories and presumes an audience familiar with evaluation research and its discontents. The debate about evaluation being a science and discipline is only dealt with obliquely or left as implied. Arguably, too much space is devoted to complexity at the expense of other possible justifications for a realist perspective.

Often the challenge facing evaluation reflects a tension between those who undertake evaluations and those who are both the objects and subjects of them. In part this tension derives from efforts to cast itself as a rigorous scientific endeavour that are experienced as a disciplinary rigidity that yield evaluations that are less than useful to those whom they are supposed to serve. Rather than being more reflexive endeavour – meeting policies and programs where they are at, in order to help them get to where they might be – a too narrowly circumscribed evaluation theory and practice make it only slightly less onerous than trial by ordeal. By introducing a realist perspective to evaluation research, Pawson hopes to rectify this situation.

A witty, at times, irreverent critique of evaluation research as a science and discipline, The Science of Evaluation deserves to be widely read and discussed. In a literature that often struggles to be lucid, accessible and engaging, writing with clarity, honesty and passion, Pawson thoughtfully outlines how evaluation research can embrace a realist perspective without fearing it will lose its methodological ardour.

——————————————————–

Lawrence Buhagiar teaches courses on a wide range of subjects (social theory, neuroscience, law, quantitative and qualitative research methods, and criminal justice reform). He has held a number of university research-related positions and worked as a consultant for government, private sector and non-governmental organizations. He has been published on a diverse range of topics in the British Journal of Criminology, Theoretical Criminology, Violence Against Women, Canadian Journal of Criminology & Criminal Justice, Women & Criminal Justice, and Current Issues in Criminal Justice. He can be reached at the Department of Law and Legal Studies, Carleton University: lawrence.buhagiar@carleton.ca or found on twitter: @sociologic. Read more reviews by Lawrence.