How Well Do Structural Demand Models Work? Counterfactual Predictions in School Choice
Discrete choice demand models are widely used for counterfactual policy simulations, yet their out-of-sample performance is rarely assessed. This paper uses a large-scale policy change in Boston to investigate the performance of discrete choice models of school demand. In 2013, Boston Public Schools considered several new choice plans that differ in where applicants can apply. At the request of the mayor and district, we forecast the alternatives' effects by estimating discrete choice models. This work led to the adoption of a plan which significantly altered choice sets for thousands of applicants. Pathak and Shi (2014) update forecasts prior to the policy change and describe prediction targets involving access, travel, and unassigned students. Here, we assess how well these ex ante counterfactual predictions compare to actual outcome under the new choice sets. We find that a simple ad hoc model performs as well as the more complicated structural choice models for one of the two grades we examine. However, the structural models' inconsistent performance is largely due to prediction errors in applicant characteristics, which are auxiliary inputs. Once we condition on the actual applicant characteristics, the structural choice models outperform the ad hoc alternative in predicting both choice patterns and policy relevant outcomes. Moreover, refitting the models using the new choice data does not significantly improve their prediction accuracy, suggesting that the choice models are indeed “structural.” Our findings show that structural demand models can effectively predict counterfactual outcomes, as long there are accurate forecasts about auxiliary input variables.
We thank Boston Mayor Thomas Menino and Boston Public School Superintendent Carol Johnson for authorizing this study. Boston Public Schools staff, including Kamal Chavda, Tim Nicolette, Peter Sloan, Kim Rice, and Jack Yessayan, provided essential help. We are grateful to our discussant Liran Einav for comments inspiring Section 7, Josh Angrist, Dan McFadden and Ariel Pakes for encouragement, and participants at the McFadden 80th Birthday conference and the NBER Market Design conference for input. We also thank Nikhil Agarwal, Isaiah Andrews, Steve Berry, Glenn Ellison, Drew Fudenberg, Adam Kapor, Patrick Kline, and Michael Whinston for feedback. Financial support is from the National Science Foundation under grant SES-1426566 and the W.T. Grant Foundation. Pathak is on the scientific advisory board of the Institute for Innovation in Public School Choice. The views expressed herein are those of the authors and do not necessarily reflect the views of the National Bureau of Economic Research.
Parag A. Pathak & Peng Shi, 2020. "How well do structural demand models work? Counterfactual predictions in school choice," Journal of Econometrics, .