NATIONAL BUREAU OF ECONOMIC RESEARCH
NATIONAL BUREAU OF ECONOMIC RESEARCH

The Sensitivity of Experimental Impact Estimates: Evidence from the National JTPA Study

James J. Heckman, Jeffrey A. Smith

NBER Working Paper No. 6105
Issued in July 1997
NBER Program(s):   LS

The recent experimental evaluation of the U.S. Job Training Partnership Act (JTPA) program found negative effects of training on the earnings of disadvantaged male youth and no effect on the earnings of disadvantaged female youth. These findings provided justification for Congress to cut the budget of JTPA's youth component by over 80 percent. In this paper, we examine the sensitivity of the experimental impact estimates along several dimensions of construction and interpretation. We find that the statistical significance of the male youth estimates is extremely fragile and that the magnitudes of the estimates for both youth groups are sensitive to nearly all the factors we consider. In particular, accounting for experimental control group members who substitute training from other providers leads to a much more positive picture regarding the effectiveness of JTPA classroom training. Our study indicates the value of sensitivity analyses in experimental evaluations and illustrates that experimental impact estimates, like those from nonexperimental analyses, require careful interpretation if they are to provide a reliable guide to policymakers.

download in pdf format
   (440 K)

email paper

This paper is available as PDF (440 K) or via email.

Machine-readable bibliographic record - MARC, RIS, BibTeX

Published:

Users who downloaded this paper also downloaded these:
Heckman w2861 Choosing Among Alternative Nonexperimental Methods for Estimating the Impact of Social Programs: The Case of Manpower Training
Heckman, Ichimura, Smith, and Todd w6699 Characterizing Selection Bias Using Experimental Data
Lise, Seitz, and Smith w10283 Equilibrium Policy Experiments and the Evaluation of Social Programs
Heckman and Smith The Sensitivity of Experimental Impact Estimates (Evidence from the National JTPA Study)
Heckman, Lochner, and Taber w6426 General Equilibrium Treatment Effects: A Study of Tuition Policy
 
Publications
Activities
Meetings
Data
People
About

Support
National Bureau of Economic Research, 1050 Massachusetts Ave., Cambridge, MA 02138; 617-868-3900; email: info@nber.org

Contact Us