NATIONAL BUREAU OF ECONOMIC RESEARCH
NATIONAL BUREAU OF ECONOMIC RESEARCH

How Well Do Structural Demand Models Work? Counterfactual Predictions in School Choice

Parag A. Pathak, Peng Shi

NBER Working Paper No. 24017
Issued in November 2017
NBER Program(s):ED, IO, LS

Discrete choice demand models are widely used for counterfactual policy simulations, yet their out-of-sample performance is rarely assessed. This paper uses a large-scale policy change in Boston to investigate the performance of discrete choice models of school demand. In 2013, Boston Public Schools considered several new choice plans that differ in where applicants can apply. At the request of the mayor and district, we forecast the alternatives' effects by estimating discrete choice models. This work led to the adoption of a plan which significantly altered choice sets for thousands of applicants. Pathak and Shi (2014) update forecasts prior to the policy change and describe prediction targets involving access, travel, and unassigned students. Here, we assess how well these ex ante counterfactual predictions compare to actual outcome under the new choice sets. We find that a simple ad hoc model performs as well as the more complicated structural choice models for one of the two grades we examine. However, the structural models' inconsistent performance is largely due to prediction errors in applicant characteristics, which are auxiliary inputs. Once we condition on the actual applicant characteristics, the structural choice models outperform the ad hoc alternative in predicting both choice patterns and policy relevant outcomes. Moreover, refitting the models using the new choice data does not significantly improve their prediction accuracy, suggesting that the choice models are indeed “structural.” Our findings show that structural demand models can effectively predict counterfactual outcomes, as long there are accurate forecasts about auxiliary input variables.

You may purchase this paper on-line in .pdf format from SSRN.com ($5) for electronic delivery.

Access to NBER Papers

You are eligible for a free download if you are a subscriber, a corporate associate of the NBER, a journalist, an employee of the U.S. federal government with a ".GOV" domain name, or a resident of nearly any developing country or transition economy.

If you usually get free papers at work/university but do not at home, you can either connect to your work VPN or proxy (if any) or elect to have a link to the paper emailed to your work email address below. The email address must be connected to a subscribing college, university, or other subscribing institution. Gmail and other free email addresses will not have access.

E-mail:

Machine-readable bibliographic record - MARC, RIS, BibTeX

Document Object Identifier (DOI): 10.3386/w24017

 
Publications
Activities
Meetings
NBER Videos
Themes
Data
People
About

National Bureau of Economic Research, 1050 Massachusetts Ave., Cambridge, MA 02138; 617-868-3900; email: info@nber.org

Contact Us