Re-evaluating Learning

Developing countries have rapidly increased access to primary school, but the quality of education has remained low. Many children are now in school, but they are hardly learning. In India, for example, a 2007 nationwide survey by Pratham ¹, a large education nonprofit, found that 97 percent of the of-age children are in primary school, but only 51 percent of third graders could read a simple first-grade paragraph, and only 33 percent could do simple subtraction. If developing countries are to attain meaningful universal primary education, they must improve the quality of education.

This is a formidable task: for starters, rising enrollment, unaccompanied by additional budget outlays, has increased pressure on available resources. Classes in the lower grades often are very large, and the children arrive with wide-ranging levels of preparedness. These large and heterogeneous classes can challenge pedagogy. The curricula, set nationally and often inherited in large part from the colonial period, are not adapted to local challenges and needs. Too often, they presuppose competencies that many of the first-generation learners do not have. Besides these challenges, teachers face lax incentives, so teacher motivation is low: many teachers to do not come to school and even those who do come do not always teach.

What can be done to improve education quality in developing countries? My recent research suggests some answers to this question. My approach has centered on using randomized evaluations to identify the causal effects of promising education programs.

In a randomized evaluation, from the program's inception the researcher works in close collaboration with the practitioner. The program gets assigned randomly to part of the sample -- the treatment group -- which is compared to the rest, the comparison group. In recent years, there has been an explosion in research using randomized evaluations in development economics. Development economists have pioneered the use of research partnerships with non-governmental organizations (NGOs) or private companies. These partnerships often allow greater control over the research design and, increasingly often, input into the program design itself. Rachel Glennerster, Michael Kremer, and I² describe the various ways of incorporating random assignment in the evaluation design, and the practical challenges that go with it.

In "The Experimental Approach to Development Economics", Abhijit Banerjee and I³ review the evolution of the use of randomization in development economics research. Much like earlier work in labor economics, health, and education, the experimental research in development economics started with concerns over the reliable identification of program effects in the face of complex and multiple channels of causality. The central difficulty that randomization seeks to address is selection bias. When program participants are not randomly selected, their outcomes may differ systematically from those of non-participants. This makes it difficult to attribute any differences observed between participants and non-participants to the program itself. For example, schools that receive better inputs also may differ systematically from the other schools in other ways, for example in pedagogy and teacher incentives. However, when the program is randomly assigned, these initial differences even out and selection bias disappears. Experiments allow researchers to vary one factor at a time by randomly assigning the program to part of the sample, and therefore they yield internally valid estimates of program effects.

Thus, in the mid-1990s, development economists started doing experiments to answer basic questions about the education production function: Does better access to inputs (textbooks, teachers) affect school outcomes (attendance, test scores) -- and if so, by how much? The motivating theoretical framework was very simple, but the results were surprising. For example: Glewwe, Kremer, and Moulin⁴ found that lowering the student-textbook ratio from 4 to 2 had no effect on average test scores. Banerjee, Jacob, and Kremer ⁵ found that halving the student-teacher ratio also had no effect on test scores. These negative results prompted new reflection on the barriers to education in poor countries: If simply providing inputs does not increase the quality of education in poor countries, then it must be necessary to change the organization of teaching in schools, both the pedagogy and the incentives faced by students and teachers. This led to a new round of field experiments motivated by the general question: Can changing the organization of teaching in schools affect education outcomes? For the most part, these more recent projects have varied more than one factor at a time in different experimental groups, making randomization a powerful tool for examining the role of incentives, spillovers, and other key questions in the economics of education.

I have contributed to this literature with four projects.

Remedying education

One finding of Glewwe, Kremer, and Moulin ⁶ was that, while the average child did not benefit from textbooks, students who were already proficient did benefit. A possible explanation for this, the authors conclude, could be that the textbook (and the curriculum) was too advanced for the majority of the students. Motivated by such evidence, my research first examined programs that seek to teach students what they can learn, rather than what a centrally set curriculum says they should learn.

In the first of these projects Abhijit Banerjee, Shawn Cole, Leigh Linden, and I ⁷ evaluated a remedial education program in urban India. The nonprofit Pratham hired locals with some secondary education, trained them for two weeks, and deployed them to local schools as teacher's aides specializing in remedial instruction. The remedial curriculum targeted students in grades three and four who did not have first-grade math and reading competencies. These students were pulled out of the regular classroom and worked with the teacher's aide for half the four-hour school day. Test scores in this group increased by 0.6 standard deviations, a large effect.

The second project replicated this finding in a very different context. Abhijit Banerjee, Rukmini Banerji, Rachel Glennerster, and I ⁸ evaluated Read India, another remedial education program. Pratham gives rural volunteers (educated youth from the village) a week's training in its reading pedagogy and deploys them back to their villages to run after-school reading programs. We found that after a year, among students who could not read at baseline, those who participated in Read India were 60 percentage points more likely to be able to recognize letters than those in comparison villages. The findings already have affected policy: based on this demonstrated effectiveness, Pratham secured funding from the Gates and Hewlett foundation to extend the Read India to 100 districts, covering millions of children. And so, even when the instructor has no formal teacher's training, remedial education focusing on what children need to know to take advantage of the available inputs can be

There are two main potential explanations for these results. First, the remedial instruction, by focusing on what students do not know rather than the inappropriate curriculum, allows them to learn more effectively. Second, the teachers hired by Pratham were particularly motivated. Because the remedial instruction was always delivered by the potentially more-motivated teacher, we cannot distinguish the relative importance of these two factors.

Yet disentangling the relative importance of these two mechanisms is key for effective policy design, because nothing constrains them a priori to be embodied in the same program. For example, many more marginalized children could be taught basic competencies if the regular teachers were trained and instructed to focus on them. Conversely, more motivated teachers could teach the standard curriculum to all the children, if motivation were the salient factor.

Reorganizing the classroom

Thus, a third project, conducted in rural Kenya, was set up to assess the importance of the two factors; Pascaline Dupas, Michael Kremer, and I ⁹ designed the experiment. When Kenya introduced free primary education in 2003, class sizes exploded in the lower grades. At the beginning of the program, in 2005, the average first-grade class in the area where we worked was 83 students, and in 28 percent of the classes it was more than 100. The program provided funds, starting in the second term, to 140 schools, randomly selected out of 210 possibilities, to hire extra teachers on one-year renewable contracts. (The extra teachers were fully qualified but young and inexperienced, being recent teacher's college graduates.) In 121 of the 140 program schools, there was just one first-grade class. These classes were split into two sections. In 60 randomly selected schools, students were quasi-randomly assigned to sections; in the remaining 61, students were ranked by prior achievement (first-term grades) and the top a bottom halves were assigned to different sections. In all 121 schools, the teachers were randomly assigned to sections from a common pool of extra and regular teachers.

We compared test scores in 61 tracking schools and 60 non-tracking schools after 18 months and found that students in tracking schools scored 0.14 standard deviations higher on average, regardless of their initial score. This suggests that students benefit from being taught in more homogenous peer groups. We argue that greater homogeneity allowed teachers to tailor their teaching to what the students did not know. We found, for example, that students assigned to the bottom section seemed to gain most in the easier competencies and least in the hardest competencies.

We also found, however, that compared to those assigned to regular teachers, students assigned to the extra teacher have significantly (0.18 standard deviation) higher test scores, both in tracking and non-tracking schools. There were other differences between these two groups-for example, students assigned to the extra teacher were more likely to always be taught by the same teacher, whereas the regular teachers often adopted a rotation system by which different teachers teach different subjects. Even so, the test-score difference does suggest that motivation is important. The young and inexperienced but highly motivated teacher seems to be more effective than several experienced but unmotivated teachers put together.

Thus, the findings suggest that both pedagogy and incentives matter-ability to adapt what is taught in the classroom to what the students can learn benefits everyone, but teacher motivation makes a difference as well. The findings also confirm that just increasing inputs, without any other changes, is not effective: students who were assigned to the regular teacher in non-tracking schools did not perform significantly better than students in comparison schools.

Restructuring teacher incentives

So, teacher motivation matters, but how can teachers be incentivized? One possibility is to reward teachers for improved test scores. But, as studies in the United States suggest, this can lead to teachers focusing on the proximal (rewarded) outcome, rather than the ultimate (policy target) outcome. In particular, teachers can focus on acing the test, rather than learning the curriculum. Glewee, Ilias, and Kremer ¹⁰find, for example, that when teachers in Kenya were offered such rewards, test scores rose in the short term. Because the test-score gains did not persist, the authors suggest that the teachers may have been "teaching to the test."

Another possibility is to reward teacher effort directly-if it can be observed. In developing countries, there is a significant margin of improvement in one relatively easy-to-observe dimension of teacher effort, namely, the amount of time the teacher spends in front of the classroom. The Kenya tracking study also found that teachers who face strong incentives do come to school regularly: the teachers hired on short contracts were more likely to be in school during random checks than the regular teachers. It seems relatively easy to monitor teacher presence, so would penalizing chronic absence (or rewarding presence) improve teacher presence and learning?

A priori, it is not evident that direct attendance-based teacher incentives would improve learning. Teachers could always come to school but not teach: in the Kenya tracking study, only 54 percent of the regular teachers (compared to 84 percent of the extra teachers) in school on a given day were teaching in the classroom, the rest being in the teacher's room. And, in a five-country study, Chaudhury et al¹¹ found that 19 percent of teachers were absent and only half of those present actually were teaching at the time of the unannounced visit.

Thus, to address this empirical question, in a fourth project, Rema Hanna and I¹² evaluated the impact of direct, attendance-based incentives on teacher presence, and student learning. The NGO Seva Mandir runs single-teacher schools in remote rural Rajasthan, India. The teachers were given durable cameras with date and time functionality and asked to photograph themselves with the children at the beginning and at end of each school day. Attendance was determined based on the number of valid photographs and the teacher's pay was based on attendance. Not surprisingly, the teacher presence increased. Chronic absence fell from 40 percent to 20 percent. What's more, there is no evidence that when they were in school the teachers were less likely to teach or that they taught differently. With teaching time increased, test scores increased by 0.17 standard deviations. This suggests that direct, attendance-based incentives-applied systematically-can improve learning.

Re-empowering the parents?

It may be more difficult, though, to apply such incentives on teachers already in government service. They are politically empowered and they are accustomed to lax enforcement of incentive structures. On paper, the teachers answer to the government which answers to the parents. Many international organizations, such the World Bank, have argued that one way to strengthen teacher incentives is to empower the parents and to get them involved in the schools. Parents, the argument goes, can monitor teachers better and they are more motivated to improve school quality than faraway government officials; increasing their awareness of poor school quality, through information, and empowering them to do something about it, by increasing their control of school resources, should lead to improvements in school quality.

A finding from the second project suggests caution. Alongside our evaluation of Read India, my co-authors Abhijit Banerjee, Rachel Glennerster, Rukmini Banerji, and I¹³ also examined the impact of providing parents with information on learning levels and on the resources available to them to change their school. Despite days spent in villages conducting meetings, to get parents to effectively engage with the school system and teachers to change their behavior, the information and mobilization campaign had no effect. If confirmed in further research, this finding would suggest that, in the short run, governments should retain the responsibility of getting the schools to work for poor people.

Re-evaluating Learning-a summing up

Together, a series of randomized evaluations of education programs in developing countries have taught us something about how education in developing countries can be improved: focus teaching on skills students need to progress further; find ways to motivate teachers. Neither of these is necessarily an easy, ready-to-implement prescription. Much more work is needed to develop programs that can achieve these two objectives on a large enough scale, especially given the political economy of education in developing countries. While neither suggests plug-and-play prescriptions, they do give us ample direction about where to search.

What's more, these experiments have also taught us something about how to search, how we can learn about learning. Each experiment answers some questions and asks new ones; the next study builds on the previous one, progressively suggesting a model of education which is ready to be enriched over time.

^1. Pratham Resource Center, Annual State of Education Report, Mumbai, India: Pratham Resource Center, January 2008.

^2. E. Duflo, R. Glennerster, and M. Kremer, "Using Randomization in Development Economics: A toolkit," NBER Technical Working Paper No. 333, December 2006.

^3. A. Banerjee and E. Duflo, "The Experimental Approach to Development Economics," NBER Working Paper No. 14467, November 2008.

^4. P. Glewwe, M. Kremer, and S. Moulin, "Many Children Left Behind? Textbooks and Test Scores in Kenya," NBER Working Paper No. 13300, August 2007; forthcoming in American Economic Journal: Applied Economics.

^5. A. Banerjee, S. Jacob, and M. Kremer, with J. Lanjouw and P. Lanjouw, "Moving to Universal Education! Costs and Tradeoffs," MIT mimeo, 2005.

^6. P. Glewwe, M. Kremer, and S. Moulin, "Many Children Left Behind? Textbooks and Test Scores in Kenya."

^7. A. Banerjee, S. Cole, E. Duflo, and L. Linden, "Remedying Education: Evidence from Two Randomized Experiments in India," NBER Working Paper No. 11904, December 2005, and Quarterly Journal of Economics, 122 (3) (August 2007), pp. 1235-64.

^8. A. Banerjee, R. Banerji, E. Duflo, R. Glennerster, and S. Khemani, "Pitfalls of Participatory Programs: Evidence from a randomized evaluation in education in India," NBER Working Paper No. 14311, September 2008.

^9. E. Duflo, P. Dupas, and M. Kremer, "Peer Effects and the Impact of Tracking: Evidence from a Randomized Evaluation in Kenya," NBER Working Paper No. 14475, November 2008.

^10. P. Glewwe, N. Ilais, and M. Kremer, "Teacher Incentives," NBER Working Paper No. 9671, May 2003; a substantial revision is P. Glewwe, N. Ilais, and M. Kremer, "Teacher Incentives," Harvard University mimeo, June 2008.

^11. N. Chaudhury, J. Hammer, M. Kremer, K. Muralidharan, and F. H. Rogers, "Teacher Absence in India: A Snapshot," Journal of the European Economic Association, 3:2-3 (April-May 2005), pp. 658-67.

^12. E. Duflo and R. Hanna, "Monitoring Works: Getting Teachers to Come to School," NBER Working Paper No. 11880, December 2005.