Incentives Matter in Schools

This short summary describes our research on educator incentive programs in the Dallas Independent School District (DISD). Minh Nguyen, Ben Ost, and Andrew Morgan are coauthors on both studies completed for this project, and Jin Luo and Ayman Shakeel are coauthors on one of the studies.

Past studies consistently show the substantial impact of effective teachers at raising students’ achievement and their future earnings.¹ Yet, as US schools search for approaches to ameliorate pandemic learning losses and to improve overall student performance, this general research finding has had little influence on efforts to improve outcomes in the vast majority of districts and states. In fact, there is little consensus about effective school improvement policies that can be taken to scale, particularly for the most disadvantaged schools. Importantly, our research on the dramatic personnel reforms introduced in the DISD suggests that standard economic solutions that recognize the importance of incentives and working conditions can provide a basis for a revitalized education system.

The DISD story begins when Mike Miles was appointed DISD superintendent in 2012 and immediately embarked on an overhaul of the long-standing evaluation and pay systems that gave high marks to most educators and produced little relationship between job performance and compensation. The district introduced two sets of reforms to its personnel system—the Principal Excellence Initiative (PEI) in 2013, and the Teacher Excellence Initiative (TEI) in 2015. Both employed new evaluation systems and revised pay schedules to match the evaluations. The TEI was more radical because it replaced a rigid system that based salaries on teacher experience and teacher degree level, two factors that bear limited relationship to student outcomes. Principal contracts were always more flexible than those for teachers, although PEI introduced clear incentives related to student achievement.

DISD then leveraged the pay-for-performance system to offer salary inducements to attract and retain highly effective educators in the district’s lowest achievement schools. Although past attempts to use monetary incentives to attract teachers to the most disadvantaged schools had generally not led to higher student achievement, the typical program offered modest stipends that were not based on demonstrated teacher effectiveness. When DISD introduced the Accelerating Campus Excellence (ACE) program, it included stipends based on prior performance that could increase salaries by up to $10,000 per year. This induced many of the most highly rated teachers to move to and remain in the lowest performing schools in the district for as long as the bonuses lasted.

Our two studies that evaluate the effects of TEI and ACE, respectively, offer direct evidence that 1) appropriately structured incentives based on student performance can lead to significant system-wide improvement in outcomes in large urban school districts; and 2) it is possible to quickly and decisively raise the achievement of the lowest performing students with large, targeted salary inducements for highly effective educators. Within two years, average achievement in the ACE elementary schools exceeded the district average. When the incentives were removed after three years, however, many highly effective teachers left, and school performance declined substantially. Importantly, Superintendent Miles left the district in 2015, just as TEI and ACE were getting underway, so these results speak to the power of the reforms and not just to the skills and charisma of the reformer.

The Impact of District-Wide Performance Incentives

We investigate the effects of TEI on math and reading achievement in the DISD.² TEI introduced a multimeasure evaluation system that combined a student achievement component, a performance component primarily based on supervisor observations of teaching, and a survey component based on feedback from students in second grade and higher. Student achievement came from the standardized state tests, while the structured observational component involved ten 10–15 minute spot observations and one 45-minute extended observation per year. Importantly, the previously introduced PEI placed considerable weight on principal effectiveness at supporting teachers. Together, the systems incorporate features designed to mitigate arbitrary judgments by principals, reward teamwork, and support low-achieving children: TEI bases a portion of the teacher achievement component on the school average, and PEI penalizes principals whose subjective evaluations of teachers diverge from student achievement growth and rewards the closing of the achievement gap.

The TEI score constitutes the primary determinant of teacher salary with some downward protections for incumbent educators at the time of the reform, limits on movement among categories, and additional requirements for reaching the highest salary levels. Table 1 shows the salaries and target distribution for teachers in each category in 2015. The fixed distributions across categories protect the budget from evaluation inflation.

Estimating the effects of TEI requires identifying an appropriate comparison district. As the second largest district in Texas with almost 160,000 students in 2015, DISD has no natural comparison district in Texas. Potential comparison districts of similar size and student composition outside of Texas would not be subject to the same state education regulations and financing or take the same standardized test. This leads us to use synthetic control methods to estimate the time trend of what DISD achievement would have been in the absence of the reforms. Specifically, we construct a comparison school for each of the schools in Dallas by taking a weighted average of all other schools in Texas where at least 60 percent of students have incomes that are low enough to make them eligible for a subsidized lunch. The weights are chosen to minimize the pretreatment difference in outcomes between each DISD school and its synthetic control school for the years 2004–12. We then aggregate DISD and control schools to the district level and compare the pattern of achievement before and after the introduction of TEI in 2015.

The left panel of Figure 1 illustrates the substantial effect of TEI on district-wide math achievement that increases to approximately 0.1 standard deviation by 2019. This is an educationally and economically large effect that compares favorably with expensive interventions including the reduction of elementary school class size. Importantly, Dallas and the synthetic control district have similar math scores in 2013 and 2014, years not used in the matching algorithm. This provides evidence of common pretreatment trends in the DISD and comparison schools that supports the validity of the comparison groups.

Any potential comparison schools in suburban or rural communities could be problematic if schools in these districts experience systemically different economic, social, and policy shocks than those in large Texas districts including DISD. We therefore assess the robustness of our estimates to restricting the donor pool to the largest districts and find that the treatment effects increase as the donor pool becomes more restrictive. The 2019 coefficient increases from 0.1 with the unrestricted donor pool of schools from high-poverty districts to 0.18 with a donor pool of schools from the 10 largest high-poverty districts.

The right panel of Figure 1 reveals a different pattern for reading with smaller and statistically insignificant effects. Effect sizes for reading do increase and become significant by 2019 as the donor pool becomes more restrictive, but the insignificant effects in the less restricted donor pools lead us to be cautious in making any claim regarding reading improvement. Note that the larger impact of schools on math compared to reading is consistent with prior evidence, a result often attributed to parents having a greater impact on reading than on math achievement.³ But it may also reflect larger variation in teacher effectiveness at math as opposed to reading instruction.

TEI strengthens incentives for current teachers, provides abundant information for teacher development, and makes the district a more desirable place for more effective teachers who earn higher salaries, all of which can contribute to the growth in achievement. We find that most of the improvement in math scores results from changes in the composition of the DISD teaching force. Those teachers who leave the district are, on average, substantially less effective than the peers who remain.

Improving Poorly Performing Schools

A wide range of programs designed to improve the quality of instruction for disadvantaged students at traditional public schools have been implemented, but the overall record has been underwhelming. Moreover, when programs appear successful, it is difficult to extract what elements are most important. Dallas addressed the challenge of improving its poorest performing schools by using financial incentives to attract and retain highly effective educators in schools with the lowest student achievement. Our analysis finds that the ACE program produced dramatic and lasting achievement increases.⁴ 

ACE provided annual stipends for moving to one of the ACE schools ranging from $6,000 for a Progressing teacher to $10,000 for an Exemplary or Master teacher. The program was initially rolled out in four elementary schools in 2016 (ACE 1) and then an additional five elementary schools in 2018 (ACE 2). In 2019, DISD scaled back the intervention for three of the four ACE 1 elementary schools with the fourth being assigned to the new ACE cohort. The ACE program replaced all the principals and required all prospective teachers for the targeted schools, including existing teachers, to apply for jobs.

The district selected the lowest performing schools in 2014 to receive the ACE treatment in 2016, and we use the next lowest performers as controls. Although control group schools initially performed slightly better than the ACE schools, they followed similar pretreatment trends.

A common narrative about US schools is that teachers do not respond to incentives, but this is not what we observe in the DISD. The left panel of Figure 2 shows that ACE substantially improved the distribution of teacher effectiveness, measured in the year prior to ACE implementation. Before the program, teacher ratings in the soon-to-be ACE schools are concentrated in the lower part of the distribution, but following program adoption and replacement of roughly 80 percent of the teachers, the distribution of teacher ratings in the ACE schools based on their effectiveness measured in the same pretreatment year moves sharply to the right. By comparison, the distribution of teacher ratings in control schools barely changes over the period (right panel).

Shifts in the teacher evaluation distribution do not constitute direct evidence of the success of ACE. We adopt a difference-in-differences approach that compares the performance of ACE schools to that of the control schools to estimate the effects of ACE on achievement. 

What happened to student achievement? Figure 3 plots the estimated effects of ACE on math and reading achievement, showing large treatment effects of around 0.5 standard deviation in math and 0.35 standard deviation in reading for 2016–18, the three program years. This takes achievement in the ACE schools from the bottom of the DISD distribution in 2015 to the district average in math and slightly below the district average in reading.

The subsequent substantial declines in math and reading achievement in 2019 show clearly the importance of the salary stipends linked with effectiveness. Because the initial cohort of ACE schools did so well, most of the monetary rewards for teachers were eliminated in 2019. Almost half of the more effective teachers in the highest rating categories left the ACE schools after 2018, with most moving to another DISD school. By comparison, over 80 percent had remained in an ACE school following the 2017 school year. Only 10 percent of the lower-rated teachers left the ACE school following the loss of stipends.

When we follow ACE students out of their elementary schools into middle schools, the event-study estimates suggest that students who were in ACE elementary schools for at least two years had substantially higher middle school math achievement. This indicates that the impacts reflected true foundational learning rather than strategically teaching to the test.

Although a cost-benefit analysis is difficult to do with precision, our estimates indicate that the $1,000 of stipend costs for the ACE program return some 17 times that cost in terms of student lifetime earnings. Alternatively, the ACE expenditures yield substantially greater achievement gains than existing estimates for class size reductions or for high-dosage tutoring.

The ACE intervention also included additional instruction time, an enhanced after-school program, and some other components, but 85 percent of program costs went towards the salary enhancements. The impacts for the second cohort of ACE schools treated in 2018 are quite similar. We nonetheless focus on the first cohort because we can observe students following their entry into middle school and observe schools post-treatment.

Some Conclusions and Interpretations

Our evaluations suggest that the DISD personnel reforms had a dramatic effect on the quality of instruction and achievement. 

Having a credible evaluation system in place permitted the introduction of performance-based incentives to get highly effective teachers to teach in the lowest performing schools. Contrary to a general pessimism about the possibility of improving schools serving the most disadvantaged students in major cities, this policy demonstrates that teachers respond to incentives and that the most effective teachers can turn around the lowest performing schools.

A common argument against this type of incentive program is one of political resistance that introduces inertia into school institutions, making it difficult to produce meaningful changes. For example, Race to the Top (RTT)—the federal initiative to expand high-stakes teacher evaluation—failed to induce the adoption of high-stakes reforms that included meaningful salary differences connected with performance. The design of the incentive structure incentive structure likely contributed to its lack of success in raising the quality of instruction and achievement.⁵

Incentives provide a possible answer to the general problem of resistance to substantial institutional change. The Texas legislature, impressed by early results from Dallas, introduced a state incentive program in 2019 that provides grants to districts that are willing to have evaluations based on student achievement and that assigns effective teachers where they are most needed, a more demanding standard than that included in RTT. By 2025, more than half of Texas districts had applied for state funds to provide effectiveness-based rewards to teachers.⁶ An evaluation of teacher effectiveness and achievement under this Texas incentive program will provide crucial information for the crafting of improved legislation in the future.