Using Lagged Outcomes to Evaluate Bias in Value-Added Models
Value-added (VA) models measure the productivity of agents such as teachers or doctors based on the outcomes they produce. The utility of VA models for performance evaluation depends on the extent to which VA estimates are biased by selection, for instance by differences in the abilities of students assigned to teachers. One widely used approach for evaluating bias in VA is to test for balance in lagged values of the outcome, based on the intuition that today’s inputs cannot influence yesterday’s outcomes. We use Monte Carlo simulations to show that, unlike in conventional treatment effect analyses, tests for balance using lagged outcomes do not provide robust information about the degree of bias in value-added models for two reasons. First, the treatment itself (value-added) is estimated, rather than exogenously observed. As a result, correlated shocks to outcomes can induce correlations between current VA estimates and lagged outcomes that are sensitive to model specification. Second, in most VA applications, estimation error does not vanish asymptotically because sample sizes per teacher (or principal, manager, etc.) remain small, making balance tests sensitive to the specification of the error structure even in large datasets. We conclude that bias in VA models is better evaluated using techniques that are less sensitive to model specification, such as randomized experiments, rather than using lagged outcomes.
We thank Gary Chamberlain, Guido Imbens, Patrick Kline, and Jesse Rothstein for comments. On May 4, 2012, Chetty was retained as an expert witness by Gibson, Dunn, and Crutcher LLP to testify about the importance of teacher effectiveness for student learning in Vergara v. California, a case that was decided on June 10, 2014, before research on this paper began. On June 13, 2015, Friedman was retained as a potential expert witness by the Radey Law Firm to advise the Florida Department of Education on the importance of teacher effectiveness for student learning, as related the development of Draft Rule 6A-5.0411. His work on this case lasted for approximately one week and he had no involvement in any legal proceeding. On December 11, 2015, Friedman was retained as an expert witness by the Houston Independent School District in the matter of Houston Federation of Teachers, et al. v. Houston Independent School District. Friedman has not shared or discussed this paper with the parties in that case or their attorneys. This research was funded in part by the National Science Foundation.
Raj Chetty & John N. Friedman & Jonah Rockoff, 2016. "Using Lagged Outcomes to Evaluate Bias in Value-Added Models," American Economic Review, vol 106(5), pages 393-399. citation courtesy of