Counting what counts – the art and science of measurement in development: This was the title of Claire Hutchings’ compelling talk for the Department of International Development’s Cutting Edge Issues in Development Thinking and Practice series last Friday, 25 January. Read about her lecture and what students thought about it here.
In her lecture: Counting what counts – the art and science of measurement in aid and development, Claire Hutchings of Oxford Policy Management, offered some insights into the perennial questions of every development student and practitioner; how to understand what drives change and how to measure impact.
All too often, as many of those who have worked in the field of international development will have seen, planning and implementation are treated as separate phases of development programmes. Hutchings’ lecture highlighted the importance of moving beyond such boundaried thinking, and emphasised the need to act with intention, curiosity and humility when approaching development work. To do this, she advocates for a stronger emphasis on evidence-based learning, and testing assumptions through prototyping throughout development programmes (to see more on the importance of prototyping, you can watch this TED talk).
Monitoring and evaluation in development and humanitarian organisations can tend to focus on reporting outcomes and indicators. In part, this is a result of how accountability is currently structured, with a strong emphasis on measuring outcomes and outputs, rather than an emphasis on evaluating impact. Hutchings believes that the importance of evaluating impact, as well as knowledge generation is not prioritised as strongly as it should be in the development field, but this realization is gradually spreading among development organisations and donors, as recent initiatives from DFID and others testify. Emphasis on intention and curiosity also means trying to unpack the layers of causality. This in turn enables you to see where the project is leading to positive change, and shows what works, and what can be scaled.
To illustrate the methodology of measuring impact in more detail, Hutchings used the example of women’s empowerment, an area which is hard to measure and difficult to quantify. Drawing on her experience working at Oxfam GB, Hutchings highlighted some of the challenges of designing a measurement framework that could evaluate a complex, conceptual and contextually specific idea such as women’s empowerment. Hutchings provided some insight into how the process of design was undertaken, where Oxfam used frameworks such as transformative women’s leadership and research on power to create a tool and framework which can measure personal, relational and environmental change. The survey has questions on areas such as women’s capacity to make decisions, reproductive rights, or financial independence. Hutchings emphasised the advantage of this approach is the flexibility it gives in being able to adapt this to different contexts, or even to different definitions of gender and empowerment.
While it might be hard to put into practice some of these ideas if working for a smaller organisation, Hutchings’ lecture illustrated that the art and the science of measurement is of crucial importance to modern-day development interventions and will be even more so in the future. Hutchings’ passion for her field of work was evident throughout, and I think everyone left the room feeling a little more inspired to persevere harder in the seemingly eternal quest to measure impact.
Malika Giles (@Malika_Giles) is a student on the International Development and Humanitarian Emergencies programme, and previously worked in Tajikistan and Afghanistan. She is interested in community preparedness and state response to natural disasters, as well civil society in post conflict states.
“The first thing to know about the future is to know about the present,” aptly quoted by Claire Hutchings in her lecture on the art and science of measuring development in the Cutting Edge series last Friday. Any understanding of our future must be based on a clarity of where we are today, which derives itself from a process of revising our assumptions and being open to change in any given practice.
Claire who presently works for Oxford Policy Management emphasised the importance of applying a sound measurement framework; one that is not guided by the expected end results or a rigid set of assumptions but is instead open to the theory of change. The theory of change requires one to go through the process of continuous testing, learning and improving – more specifically, being engaged in an iterative process of learning and revising the initial hypothesis which we started off with. According to Claire, unpacking causality could be nothing short of miracles in the middle if one tried to achieve expected end outcomes without revising the initial designs or assumptions. She explained how the focus shifts away from understanding the underlying “if-then” mechanism by using an example of a log-frame. A log-frame is a planning tool that takes the shape of a matrix defining the goals, activities, and expected outcomes. This tool often ends up becoming an end in itself with people simply trying to fill in the boxes and implement them without further reflection. Very rarely do researchers go back and check if these assumptions are really playing out as they should. It is thus in circumstances like these that a theory of measurability becomes extremely important.
She also observed that another need for a theory of measurability is that a lack of measurability would make it difficult to attribute causality to its correct cause. And since change isn’t linear either, it becomes important from a policy formulation perspective to understand if it was our own intervention or other causes that could have led to an outcome.
But how does one measure something that’s immeasurable or hard to define? For instance, women’s empowerment could mean different things in different communities and there are not one, but many indicators (with varying levels of relevance) to capture it. Claire illustrated how this was measured using a continuous evaluation process in the dairy industry in rural Pakistan. The measurement approach was tailored to the rural context, involved participatory methods, and the index was revised over the course of the study which culminated in a ‘How To Guide’ on Measuring Women’s Empowerment (which continues to grow and evolve).
Thus, her key takeaways for ensuring the success of a good measurement exercise in development could be achieved by keeping the following in mind:
- Acting with intention – Having a theory of change mindset not just in the beginning of the programme, but in its entirety
- Acting with curiosity – Focusing on what we need to know instead of trying to ask what we could measure
- Acting with humility – Being able to learn from people and situations and being open to change. A willingness to reconstruct the theory of change and updating prior information.
Nilanjana Bargotra is enrolled in the MSc Development Management programme. She’s a former graduate from the Madras School of Economics (MSc Applied Quantitative Finance) and the University of Delhi (BA Economics Hons). Her primary areas of interest include understanding south-to-south foreign aid relationships and government incentive structures, and also the intersection between gender, informal economies and social entrepreneurship in the global south. Before coming to LSE, she worked as a data scientist in a beer company (Anheusur Busch InBev, also the world’s largest beer company).
Next up in the series: On Friday, 1 February, the new director of OXFAM, Danny Sriskandarajah, will give a Q&A session entitled “I’m the new director of OXFAM – ask me anything!” The session takes place from 4-6pm in the Sheikh Zayed Theatre in the New Academic Building. The series is open to LSE Students and Faculty. Other external guests should visit https://bit.ly/2AxvWVG to reserve a seat.
The views expressed in this post are those of the author and in no way reflect those of the International Development LSE blog or the London School of Economics and Political Science.