NATIONAL BUREAU OF ECONOMIC RESEARCH
NATIONAL BUREAU OF ECONOMIC RESEARCH

Measuring Test Measurement Error: A General Approach

Donald Boyd, Hamilton Lankford, Susanna Loeb, James Wyckoff

NBER Working Paper No. 18010
Issued in April 2012
NBER Program(s):   ED

Test-based accountability including value-added assessments and experimental and quasi-experimental research in education rely on achievement tests to measure student skills and knowledge. Yet we know little regarding important properties of these tests, an important example being the extent of test measurement error and its implications for educational policy and practice. While test vendors provide estimates of split-test reliability, these measures do not account for potentially important day-to-day differences in student performance.

We show there is a credible, low-cost approach for estimating the total test measurement error that can be applied when one or more cohorts of students take three or more tests in the subject of interest (e.g., state assessments in three consecutive grades). Our method generalizes the test-retest framework allowing for either growth or decay in knowledge and skills between tests as well as variation in the degree of measurement error across tests. The approach maintains relatively unrestrictive, testable assumptions regarding the structure of student achievement growth. Estimation only requires descriptive statistics (e.g., correlations) for the tests. When student-level test-score data are available, the extent and pattern of measurement error heteroskedasticity also can be estimated. Utilizing math and ELA test data from New York City, we estimate the overall extent of test measurement error is more than twice as large as that reported by the test vendor and demonstrate how using estimates of the total measurement error and the degree of heteroskedasticity along with observed scores can yield meaningful improvements in the precision of student achievement and achievement-gain estimates.

You may purchase this paper on-line in .pdf format from SSRN.com ($5) for electronic delivery.

Information about Free Papers

You should expect a free download if you are a subscriber, a corporate associate of the NBER, a journalist, an employee of the U.S. federal government with a ".GOV" domain name, or a resident of nearly any developing country or transition economy.

If you usually get free papers at work/university but do not at home, you can either connect to your work VPN or proxy (if any) or elect to have a link to the paper emailed to your work email address below. The email address must be connected to a subscribing college, university, or other subscribing institution. Gmail and other free email addresses will not have access.

E-mail:

Machine-readable bibliographic record - MARC, RIS, BibTeX

Document Object Identifier (DOI): 10.3386/w18010

Users who downloaded this paper also downloaded these:
Cascio and Staiger w18038 Knowledge, Tests, and Fadeout in Educational Interventions
Dee w17990 School Turnarounds: Evidence from the 2009 Stimulus
Grissom, Kalogrides, and Loeb w18568 Using Student Test Scores to Measure Principal Performance
Bond and Lang w19243 The Black-White Education-Scaled Test-Score Gap in Grades K-7
Levitt, List, Neckermann, and Sadoff w18165 The Behavioralist Goes to School: Leveraging Behavioral Economics to Improve Educational Performance
 
Publications
Activities
Meetings
NBER Videos
Data
People
About

Support
National Bureau of Economic Research, 1050 Massachusetts Ave., Cambridge, MA 02138; 617-868-3900; email: info@nber.org

Contact Us