xtreg, tsls and their ilk are good for one fixed effect, but what if you have more than one? Possibly you can take out means for the largest dimensionality effect and use factor variables for the others. That works untill you reach the 11,000 variable limit for a Stata regression. An attractive alternative is -reghdfe- on SSC which is an iterative process that can deal with multiple high dimensional fixed effects. A regression with 60,000 and 25,000 catagories in two separate fixed effects took 4,900 seconds on a test dataset with 100 million observation (limited to 2 cores). So it is very practical. See Abowd, Creecy and Kramarz for more information about the statistical properties..
I recently received a message From Sergio Correia with some information about a recent revision to the -reghdfe- command.
Hi Daniel, Just wanted to share with you some results about the latest version of the -reghdfe- command. I ran some benchmarks on the last version (v3.0), available at github, and found that it's actually 3-4 times as fast as areg and xtreg even for one fixed effect. As seen in the benchmark do-file (ran with Stata 13 on a laptop), on a dataset of 100,000 obs., areg takes 2 seconds., xtreg_fe takes 2.5s, and the new version of reghdfe takes 0.4s Without clusters, the only difference is that -areg- takes 0.25s which makes it faster but still in the same ballpark as -reghdfe-. All results are robust to changing the size of the dataset and the number of allowed cores, although for datasets below 1000 observations, the cost of compiling the mata files makes reghdfe slower than the built-in alternatives. Also, since I'm using a different estimation algorithm, the speed improvements when using 2 or more FEs, are even larger. Betas will be consistent except in the most extreme cases such as having one fixed effect for each observation. Sure, the estimate for the intercept of a particular individual may not be consistent, but most people care about the betas. In fact, -xtreg- doesn't even give you point estimates for the FEs, only -areg- does. Best, Sergio PS: By the way, several of the other pain points mentioned in your Stata guide, such as separate slope coefficients, fast IV with fixed effects, multiway clustering, etc. can also be solved efficiently with reghdfe
From email@example.com Wed Dec 12 18:24:23 2018 Date: Wed, 12 Dec 2018 18:24:03 -0500 From: Sergio Correia To: Ezra Karger , Daniel Feenberg Subject: Re: question about installing ftools for reghdfe Hi Ezra and Daniel, You are right on your intuition, but the memory usage can be reduced drastically at only a small speed cost. First, more details on what's going on: Most of the memory cost is independent on whether you are demeaning wrt. 1, 2, or 10 sets of fixed effects, and has most to do with the number of covariates. First, I'll try to explain what's the point of highest memory usage. What reghdfe does is the following: 1. Before running, you have a dataset in memory that will remain there (so if the dataset is 10gb, that's 10gb used from the get-go) 2. reghdfe loads the data into Mata, which can potentially double the memory usage if all covariates are double. If the covariates are instead stored as byte (which saves memory), the memory increase will be larger b/c Mata only supports double types. 3. reghdfe does "y = partial_out(y)". This operation will, for one instant, have two copies of the data, potentially doubling again the dataset. Thus, in this worst case scenario you will triple the amount of memory just due to the covariates being stored in Mata. On top of that you have the set of fixed effects, which adds a bit more memory. In your case, the key problem is the covariates, because the main use of reghdfe does not involve having 1000 covariates, and instead if you don't really care about those 1000 covariates you usually end up absorbing them (specially if they are dummies). Now, there are two options that I use to reduce memory usage drastically: pool(#) and compact. The pool(1) option will perform the partial on the covariates one by one, so the memory usage is 2x instead of 3x. As you increase the number to e.g. pool(5) you increase memory usage a bit but speed increases. Further, the compact option will first -preserve- the data, and then drop observations as they are loaded into Mata. Thus, the 2x memory usage is now closer to 1x memory usage. Hope this helps! Best, Sergio