NATIONAL BUREAU OF ECONOMIC RESEARCH
NATIONAL BUREAU OF ECONOMIC RESEARCH
loading...

Opportunities and Challenges: Lessons from Analyzing Terabytes of Scanner Data

Serena Ng

NBER Working Paper No. 23673
Issued in August 2017, Revised in August 2017
NBER Program(s):Technical Working Papers

This paper seeks to better understand what makes big data analysis different, what we can and cannot do with existing econometric tools, and what issues need to be dealt with in order to work with the data efficiently. As a case study, I set out to extract any business cycle information that might exist in four terabytes of weekly scanner data. The main challenge is to handle the volume, variety, and characteristics of the data within the constraints of our computing environment. Scalable and efficient algorithms are available to ease the computation burden, but they often have unknown statistical properties and are not designed for the purpose of efficient estimation or optimal inference. As well, economic data have unique characteristics that generic algorithms may not accommodate. There is a need for computationally efficient econometric methods as big data is likely here to stay.

You may purchase this paper on-line in .pdf format from SSRN.com ($5) for electronic delivery.

Access to NBER Papers

You are eligible for a free download if you are a subscriber, a corporate associate of the NBER, a journalist, an employee of the U.S. federal government with a ".GOV" domain name, or a resident of nearly any developing country or transition economy.

If you usually get free papers at work/university but do not at home, you can either connect to your work VPN or proxy (if any) or elect to have a link to the paper emailed to your work email address below. The email address must be connected to a subscribing college, university, or other subscribing institution. Gmail and other free email addresses will not have access.

E-mail:

Machine-readable bibliographic record - MARC, RIS, BibTeX

Document Object Identifier (DOI): 10.3386/w23673

Users who downloaded this paper also downloaded* these:
Lordan and Neumark w23667 People Versus Machines: The Impact of Minimum Wages on Automatable Jobs
Foster, Grim, Haltiwanger, and Wolf w23666 Macro and Micro Dynamics of Productivity: From Devilish Details to Insights
Gorodnichenko and Ng w23672 Level and Volatility Factors in Macroeconomic Data
Blau and Winkler w23644 Women, Work, and Family
Davis and van Wincoop w23671 Globalization and the Increasing Correlation between Capital Inflows and Outflows
 
Publications
Activities
Meetings
NBER Videos
Themes
Data
People
About

National Bureau of Economic Research, 1050 Massachusetts Ave., Cambridge, MA 02138; 617-868-3900; email: info@nber.org

Contact Us