TY  - JOUR
AU  - Heckman,James J.
AU  - Smith,Jeffrey A.
TI  - The Sensitivity of Experimental Impact Estimates: Evidence from the National JTPA Study
JF  - National Bureau of Economic Research Working Paper Series
VL  - No. 6105
PY  - 1997
Y2  - July 1997
UR  - http://www.nber.org/papers/w6105
L1  - http://www.nber.org/papers/w6105.pdf
N1  - Author contact info:
James J. Heckman
Department of Economics
The University of Chicago
1126 E. 59th Street
Chicago, IL 60637
Tel: 773/702-0634
Fax: 773/702-8490
E-Mail: jjh@uchicago.edu

Jeffrey Smith
Department of Economics
University of Michigan
238 Lorch Hall
611 Tappan Street
Ann Arbor, MI 48109-1220
Tel: 734/764-5359
E-Mail: econjeff@umich.edu
M1  - published as James J. Heckman, Jeffrey Smith. "The Sensitivity of Experimental Impact Estimates (Evidence from the National JTPA Study)," in David G. Blanchflower and Richard B. Freeman, editors, "Youth Employment and Joblessness in Advanced Countries" University of Chicago Press (2000)

AB  - The recent experimental evaluation of the U.S. Job Training Partnership Act (JTPA) program found negative effects of training on the earnings of disadvantaged male youth and no effect on the earnings of disadvantaged female youth.  These findings provided justification for Congress to cut the budget of JTPA's youth component by over 80 percent.  In this paper, we examine the sensitivity of the experimental impact estimates along several dimensions of construction and interpretation.  We find that the statistical significance of the male youth estimates is extremely fragile and that the magnitudes of the estimates for both youth groups are sensitive to nearly all the factors we consider.  In particular, accounting for experimental control group members who substitute training from other providers leads to a much more positive picture regarding the effectiveness of JTPA classroom training.  Our study indicates the value of sensitivity analyses in experimental evaluations and illustrates that experimental impact estimates, like those from nonexperimental analyses, require careful interpretation if they are to provide a reliable guide to policymakers.
ER  -