NATIONAL BUREAU OF ECONOMIC RESEARCH
NATIONAL BUREAU OF ECONOMIC RESEARCH

Using Lagged Outcomes to Evaluate Bias in Value-Added Models

Raj Chetty, John N. Friedman, Jonah Rockoff

NBER Working Paper No. 21961
Issued in February 2016
NBER Program(s):Children, Economics of Education, Labor Studies, Public Economics

Value-added (VA) models measure the productivity of agents such as teachers or doctors based on the outcomes they produce. The utility of VA models for performance evaluation depends on the extent to which VA estimates are biased by selection, for instance by differences in the abilities of students assigned to teachers. One widely used approach for evaluating bias in VA is to test for balance in lagged values of the outcome, based on the intuition that today’s inputs cannot influence yesterday’s outcomes. We use Monte Carlo simulations to show that, unlike in conventional treatment effect analyses, tests for balance using lagged outcomes do not provide robust information about the degree of bias in value-added models for two reasons. First, the treatment itself (value-added) is estimated, rather than exogenously observed. As a result, correlated shocks to outcomes can induce correlations between current VA estimates and lagged outcomes that are sensitive to model specification. Second, in most VA applications, estimation error does not vanish asymptotically because sample sizes per teacher (or principal, manager, etc.) remain small, making balance tests sensitive to the specification of the error structure even in large datasets. We conclude that bias in VA models is better evaluated using techniques that are less sensitive to model specification, such as randomized experiments, rather than using lagged outcomes.

download in pdf format
   (525 K)

email paper

Machine-readable bibliographic record - MARC, RIS, BibTeX

Document Object Identifier (DOI): 10.3386/w21961

Published: Raj Chetty & John N. Friedman & Jonah Rockoff, 2016. "Using Lagged Outcomes to Evaluate Bias in Value-Added Models," American Economic Review, vol 106(5), pages 393-399. citation courtesy of

Users who downloaded this paper also downloaded* these:
Chetty, Hendren, Lin, Majerovitz, and Scuderi w21936 Childhood Environment and Gender Gaps in Adulthood
Chetty, Friedman, and Rockoff w17699 The Long-Term Impacts of Teachers: Teacher Value-Added and Student Outcomes in Adulthood
Bacher-Hicks, Kane, and Staiger w20657 Validating Teacher Effect Estimates Using Changes in Teacher Assignments in Los Angeles
Chetty, Hendren, and Katz w21156 The Effects of Exposure to Better Neighborhoods on Children: New Evidence from the Moving to Opportunity Experiment
Chetty, Friedman, and Rockoff w19424 Measuring the Impacts of Teachers II: Teacher Value-Added and Student Outcomes in Adulthood
 
Publications
Activities
Meetings
NBER Videos
Themes
Data
People
About

National Bureau of Economic Research, 1050 Massachusetts Ave., Cambridge, MA 02138; 617-868-3900; email: info@nber.org

Contact Us