Following this year’s Nobel prize in Economics and the attention it has drawn to the use of RCT’s in development planning, Martin Haus and Rakesh Rajak question whether RCTs really have the potential to solve India’s learning crisis.  

This year’s Nobel prize in economics has been awarded to the three researchers Abhijit Banerjee, Esther Duflo and Michael Kremer, who are well-known for their field experiments in the form of randomised controlled trials (RCTs). But can that methodology make meaningful contributions to solving the problem of our schools failing our children? Can RCTs help us transform our government schools and thus solve the learning and equity crisis?

RCTs, in general, are useful for narrow optimisation or changes, and to make causal inferences, that is, to find out and prove what causes a change in outcomes in a specific context. For this, it is important that much stays the same. By tweaking one aspect, one reaches a scenario where business goes on as usual but with one change. This, then, allows us to quantify the impact of this one change, using treatment and control groups.

To illustrate, let us take a popular example that the Jameel Latif Poverty Action Lab (J-PAL), a research institute whose board includes both Duflo and Banerjee, has promoted for long: Teaching at the Right Level (TaRL). The methodology developed by Pratham, India’s leading NGO in the education field, is based on the following idea: children who fall behind in school cannot possibly benefit from being taught at a level far above their current abilities.

As a result, they will not be able to follow what is taught in the classroom. Therefore, TaRL will ensure that they are taught not according to their grade, but their ability. This methodology has seen very positive impact evaluations. It also ticks the boxes required to make it open to RCT-evaluations: it can serve as a standalone intervention, schools can be randomly assigned to treatment (using TaRL) and control groups (not using TaRL), it is straightforward to implement, and leads to quantifiable outcomes (learning levels) according to a standardised assessment.

Does that mean it is something governments should scale up? Will it help solve the learning crisis at large? We think the answer to the first question is somewhat more yes than no, but the answer to the second question is more of a no than a yes. Here is why:

Look at a class VIII classroom. You will have a huge variety of children at very different grade levels in their learning. Some might (and indeed many are) still struggling to start basic reading. Sitting in a class VIII classroom makes little sense – they have just fallen behind so much, they will not able to follow what is happening in the lesson. The solution then is TaRL. Ensure that the children who have fallen behind and have now reached class VIII are taught at the level they are. Taking the example above, this would mean practising basic reading. Put them together with students of class II who are at a similar level. Teach everyone at the right level.

But consider this for a minute: If everybody started at class I, with a class I level, why are some children falling behind while others are not? What is the core problem here? Is it that the classrooms are very diverse? Maybe, but overall government schools have children from rather similar (socio-economically weaker) backgrounds compared to the previous years (when there were not so many ‘low-cost’ private schools). The ASER reports show that it’s not so much individual children who are falling behind, but overall flat learning profiles. Not much learning is happening in schools.

The reasons for this are manifold. There is a systemic issue leading to teachers not teaching, in often, dilapidated classrooms. Textbooks arrive by the end of the school year (if at all), teachers are untrained and unmotivated, with single-teacher schools, no subject-teachers, no functioning support of the academic chain upstream (CRC, BRC, DIET), and much more. Children often fall behind because nothing much is happening in classrooms. If parents cannot support children and they get no tuition, they fall behind.

But if that is the issue, then how is TaRL solving the problem? Answer: It is not. Yes, it makes sense to use TaRL for a limited period of time to get children who are already in class VIII but cannot help them reach the level they need to be (which will be really difficult). But, then again, who shall do that teaching? If teachers do not teach, it does not matter whether they do it at the right or wrong level. They simply do not teach at any level.

The solution to the ‘learning crisis’ cannot possibly be TaRL.

To illustrate, take the following analogy: If children fall ill, you give them medicine which will require some effort and money. You can do that once or twice. But if they keep falling sick, it might be worth looking into what is making them sick. Yes, fixing the water supply to ensure that the water children drink is safe and clean will take more effort. Contaminations from different sources may cause illness. But there is no other way because continuously giving children medicine will not solve the issue of sick children. Only systemic improvements to the drinking water supply will solve the issue. Here is the problem: the medicine ticks the boxes for fitting into the RCT-requirements. The messy, political, systemic improvement does not. Research on the latter does exist, but is far less influential.

TaRL is medicine for children who are already sick. It tells us very little about how to ensure that children remain healthy. Even worse, the provision of this medicine often works only if provided by external suppliers. The core, mainstream system remains untouched and dysfunctional.

The problem is that the business of development is made for the provision of medicine by external suppliers, not for the messy and arduous task of fixing the drinking water supply. Fixing this might lead to involvement with vested interests who profit from the poor water quality. One might have to speak to farmers using pesticides which contaminate the water.

One might have to talk to ward members for funds to fix pipes. Or informal power brokers whose interests might be opposed to one’s own. In short: this is going to be a messy process involving power and politics as much as funds and technocratic fixes.

Solving the ‘learning crisis’ needs children to remain healthy instead of providing a one-off medicine every time 80% of the children fall sick.

Does that mean that TaRL is bad and useless? No, far from it. It is one part of the solution – an important one at that. For reforms to be successful, big results fast are important for building political and public authorisation.

The problem with the bestowing of a Nobel prize is that it fuels public misconception. While the prize winners might be well aware of the limitations, the public is not, and several organisations’ business model is what they call “evidence-based policy making”.

The problem with the domination of RCTs in development and the phrasing of “evidence-based” for interventions with “proven impact” using the “gold standard” is the depreciation of other, more relevant findings using different methodologies. Big transformations, whether of schools or economies, have not come around by putting together a basket of RCT-proven interventions, but by having a political vision and able advisers that could manoeuvre a political minefield while having an eye on possible technocratic building stones for successful reform.

The poor wording of the Nobel prize committee framing the domination of RCTs as something positive together with the public misperception around the usefulness of RCTs (which his much narrower than widely believed) make it difficult for those who advocate for messier, more uncertain, yet more relevant research for reforming public systems. Such approaches do exist. Systems thinking has led to the insight that reforming education or economic systems is more than the sum of the (reform) parts.

Techniques like Problem Driven Iterative Adaptation argue for reforming public systems led by reform teams inside ministries using regular feedback loops thus acknowledging that reforms are a journey into the unknown and that capacity gets built on the way. They thereby argue against the use of externally supplied solutions by consultants. Accounts of big transformations in the past show how governments have manoeuvred in their reform efforts.

One excellent example is Yuen Yuen Ang’s account of how China escaped the poverty trap. Jean Drèze has cautioned that an overreliance on RCTs is bad and that economists should not assume that they have any superior knowledge to advise governments. This, however, remains unheard by many institutes whose business model is to point out the “gold standard” of their favoured methodology – RCTs. This is a highly doubtful claim given that by and large, RCTs have not improved development. As eminent economist Lant Pritchett put it frankly: “The Debate about RCTs in Development is over. We won. They lost.” (“we” referring to those critical of RCTs.)

The objective of this article was to answer the questions whether RCTs can make meaningful contributions to solving the problem of our schools failing our children and whether RCTs can help us transform government schools and thus solve the learning and equity crisis.

The answer is “no, but”. RCTs have very limited use for a real transformation of our school education system. This transformation has large political and social implications, something RCTs are not made for. RCTs further have little ability to help governments navigate during reform journeys into the unknown – given the fact that systems are more than their parts.

RCTs can, however, e.g. help in understanding parts of individual behaviour to changing incentives. A useful RCT in the Indian context comes from Bihar where the fund flow of NREGA has been improved with the help of J-PAL. This is an important part of a balanced assessment of RCTs: saying that their use is largely limited to rather narrow optimisation or to choose between very specific options is not to dismiss their relevance. Small tweaks of large government programs can mean that over several years, thousands of crores either reach beneficiaries or not.

The bottom line for education secretaries and ministers: Have a look at RCTs. But consume them like good wine: do not get drunk on them. RCTs can be insightful. They are not free of bias. They are not useful when it comes to guiding one in the direction of big reform efforts. But they can and should be part of a wider methodological repertoire that helps us find out what might work. They cannot do away with the political dimension of reforms. The real world is not a lab where most things remain constant, especially during reforms.

The bottom line for those practising RCTs: Have a look at real-world reforms. Build up understanding for political priorities and the need of governments to respond to urgent and unexpected developments. This is probably something where even Nobel prize winners have room for improvement.

Try to avoid referring to your methodology as a gold standard and be honest about the limits of its use. But continue your work. We, too, despite being critical, have found many RCTs very useful to build our understanding of what or what might not work for improving our government schools.

Martin Haus has an MSc in Development Studies from the London School of Economics and is advisory board member of the Education Policy Institute of Bihar.

Rakesh K. Rajak holds a MA from the Delhi School of Social Work and is project manager at the Education Policy Institute of Bihar.

This article was first published on the thewire.in.

The views expressed in this post are those of the author and in no way reflect those of the International Development LSE blog or the London School of Economics and Political Science.