NATIONAL BUREAU OF ECONOMIC RESEARCH
NATIONAL BUREAU OF ECONOMIC RESEARCH

The Central Role of Noise in Evaluating Interventions that Use Test Scores to Rank Schools

Kenneth Y. Chay, Patrick J. McEwan, Miguel Urquiola

NBER Working Paper No. 10118
Issued in November 2003
NBER Program(s):   ED   LS

Several countries have implemented programs that use test scores to rank schools, and to reward or penalize them based on their students' average performance. Recently, Kane and Staiger (2002) have warned that imprecision in the measurement of school-level test scores could impede these efforts. There is little evidence, however, on how seriously noise hinders the evaluation of the impact of these interventions. We examine these issues in the context of Chile's P-900 program a country-wide intervention in which resources were allocated based on cutoffs in schools' mean test scores. We show that transitory noise in average scores and mean reversion lead conventional estimation approaches to greatly overstate the impacts of such programs. We then show how a regression discontinuity design that utilizes the discrete nature of the selection rule can be used to control for reversion biases. While the RD analysis provides convincing evidence that the P-900 program had significant effects on test score gains, these effects are much smaller than is widely believed.

download in pdf format
   (709 K)

email paper

This paper is available as PDF (709 K) or via email.

Machine-readable bibliographic record - MARC, RIS, BibTeX

Document Object Identifier (DOI): 10.3386/w10118

Published: Chay, Kenneth Y., Patrick J. McEwan and Miguel Urquiola. "The Central Role Of Noise In Evaluating Interventions That Use Test Score To Rank Schools," American Economic Review, 2005, v95(4,Sep), 1237-1258. citation courtesy of

Users who downloaded this paper also downloaded these:
Kane w9703 A Quasi-Experimental Estimate of the Impact of Financial Aid on College-Going
Lee and Lemieux w14723 Regression Discontinuity Designs in Economics
Chay and Greenstone w6826 Does Air Quality Matter? Evidence from the Housing Market
Hsieh and Urquiola w10008 When Schools Compete, How Do They Compete? An Assessment of Chile's Nationwide School Voucher Program
Mizala and Urquiola w13676 School Markets: The Impact of Information Approximating Schools' Effectiveness
 
Publications
Activities
Meetings
NBER Videos
Data
People
About

Support
National Bureau of Economic Research, 1050 Massachusetts Ave., Cambridge, MA 02138; 617-868-3900; email: info@nber.org

Contact Us