NATIONAL BUREAU OF ECONOMIC RESEARCH
NATIONAL BUREAU OF ECONOMIC RESEARCH

Using Lagged Outcomes to Evaluate Bias in Value-Added Models

Raj Chetty, John N. Friedman, Jonah Rockoff

NBER Working Paper No. 21961
Issued in February 2016
NBER Program(s):CH, ED, LS, PE

Value-added (VA) models measure the productivity of agents such as teachers or doctors based on the outcomes they produce. The utility of VA models for performance evaluation depends on the extent to which VA estimates are biased by selection, for instance by differences in the abilities of students assigned to teachers. One widely used approach for evaluating bias in VA is to test for balance in lagged values of the outcome, based on the intuition that today’s inputs cannot influence yesterday’s outcomes. We use Monte Carlo simulations to show that, unlike in conventional treatment effect analyses, tests for balance using lagged outcomes do not provide robust information about the degree of bias in value-added models for two reasons. First, the treatment itself (value-added) is estimated, rather than exogenously observed. As a result, correlated shocks to outcomes can induce correlations between current VA estimates and lagged outcomes that are sensitive to model specification. Second, in most VA applications, estimation error does not vanish asymptotically because sample sizes per teacher (or principal, manager, etc.) remain small, making balance tests sensitive to the specification of the error structure even in large datasets. We conclude that bias in VA models is better evaluated using techniques that are less sensitive to model specification, such as randomized experiments, rather than using lagged outcomes.

You may purchase this paper on-line in .pdf format from SSRN.com ($5) for electronic delivery.

Access to NBER Papers

You are eligible for a free download if you are a subscriber, a corporate associate of the NBER, a journalist, an employee of the U.S. federal government with a ".GOV" domain name, or a resident of nearly any developing country or transition economy.

If you usually get free papers at work/university but do not at home, you can either connect to your work VPN or proxy (if any) or elect to have a link to the paper emailed to your work email address below. The email address must be connected to a subscribing college, university, or other subscribing institution. Gmail and other free email addresses will not have access.

E-mail:

Machine-readable bibliographic record - MARC, RIS, BibTeX

Document Object Identifier (DOI): 10.3386/w21961

Published: Raj Chetty & John N. Friedman & Jonah Rockoff, 2016. "Using Lagged Outcomes to Evaluate Bias in Value-Added Models," American Economic Review, vol 106(5), pages 393-399. citation courtesy of

Users who downloaded this paper also downloaded* these:
Chetty, Hendren, Lin, Majerovitz, and Scuderi w21936 Childhood Environment and Gender Gaps in Adulthood
Chetty, Hendren, and Katz w21156 The Effects of Exposure to Better Neighborhoods on Children: New Evidence from the Moving to Opportunity Experiment
Chetty, Friedman, and Rockoff w17699 The Long-Term Impacts of Teachers: Teacher Value-Added and Student Outcomes in Adulthood
Jacob, Rockoff, Taylor, Lindy, and Rosen w22054 Teacher Applicant Hiring and Teacher Performance: Evidence from DC Public Schools
Adnot, Dee, Katz, and Wyckoff w21922 Teacher Turnover, Teacher Quality, and Student Achievement in DCPS
 
Publications
Activities
Meetings
NBER Videos
Themes
Data
People
About

National Bureau of Economic Research, 1050 Massachusetts Ave., Cambridge, MA 02138; 617-868-3900; email: info@nber.org

Contact Us