Economists and Field Research: "You Can Observe a Lot Just by Watching"

Susan Helper ¹

Forthcoming, American Economic Review, May 2000

Modern economics began with Adam Smith's visit to a pin factory, which helped him explain how the division of labor worked (1985, p. 6). However, not many economists today do much fieldwork, which involves interviews with economic actors and visits to the places they live and work. To help economists get out more, in 1994 the Sloan Foundation funded the National Bureau of Economic Research to promote field research among economists, via plant tours, conferences, and comissioned papers.

I start by discussing how fieldwork can improve economic research, drawing largely on interviews with participants in the NBER/Sloan program. I then describe techniques that can improve economists' field research. (For an expanded version of this paper, see www.nber.org/sloan).

I.Fieldwork complements other methods

Economists today typically do research using econometrics and mathematical modelling. These techniques have many strengths, but share the weakness of distance from individual economic actors. In contrast, field research allows direct contact with them, yielding several advantages:

Researchers can ask people directly about their objectives and constraints. It is not always easy to figure out someone's incentives or strategies by looking only at outcomes. For example, Jim Rebitzer wondered why many professionals complain about long hours, yet few firms offer the option of short hours. In talking with lawyers, he learned that partners found it difficult to decide whom to promote in order to maximize their incomes. "One comment that stuck with me was a partner saying about an associate, 'She does really good work, but I wonder, does she like money enough?' That is, he wanted to know, will she work really hard? " Comments by this partner and others implied that they used work hours as a proxy for this propensity to work. This insight led Renee Landers, Rebitzer, and Lowell Taylor (1996) to build a model and collect survey data which suggested that these observability constraints led to incentives to work inefficiently long hours.

Fieldwork allows exploration of areas with little pre-existing data or theory. I started my dissertation research thinking I would look at automakers' make/buy decisions. But when I started interviewing and reading trade journals, I realized that important changes--not reflected in the existing literature--were occurring on the 'buy' side. US automakers were moving from adversarial deals (in which they "would steal a dime from a starving grandmother", one supplier said) to "voice" relationships in which they worked with suppliers to improve performance. My qualitative study (1991) suggested that information exchange and commitment were important determinants of supplier performance, leading me to collect survey data on these factors. One finding was that voice relationships were associated with more cost reduction, but only if complementary policies were adopted (Helper, 1999).

Fieldwork suggested to Zucker, Darby, and Brewer (1998) that number of gene-sequence discoveries was a good proxy for intellectual capital in biotechnology; their regression results were consistent with intellectual capital being the main determinant of the location and growth of biotechnology firms.

Fieldwork facilitates use of the right data. Ichniowski, Shaw, and Prennushi (1997) used interviews to determine that steel finishing lines had homogeneous technology and that there were enough such lines to allow econometric investigation of the impacts of innovative human resource policies unconfounded by technology or industry differences. They then visited 45 plants to collect production data (interviewing managers to insure compatible measures across plants) and observe what human-resource practices were in place.

Fieldwork provides vivid images that promote intuition. Edward Lazear said of his work on the change from time-rates to piece-rates at Safelite Autoglass (1996), "It's one of my most-cited papers--I think it's because everyone can imagine those guys working harder to install windshields once they're on piece-rates, and it's an image they remember a lot more than the regression coefficients."

Because of fears about the reliability of field methods (discussed below) some economists get ideas from the field, but do not discuss their fieldwork in their published articles. But this tactic causes us to lose the vividness that is a principal benefit of fieldwork. Understanding the setting can help explain differences in findings between cases, by making clear the mechanism by which variables are linked. For example, while Lazear found that a move to piece-rates increased profits at the auto-glass installer, Richard Freeman and Morris Kleiner (1998) found that a change away from piece-rates increased profits at a shoe manufacturer. Understanding the production process at the two firms is key to making sense of these results: While both papers found that productivity was higher under piece rates, time-rates at the shoe firm facilitated the introduction of a new production process that brought reduced inventories and faster new-product introduction.

Many of these insights can be translated into the language of econometrics or theory. It is possible that economists using only these methods could have generated these insights, but in fact, they didn't. Fieldwork offers a new source of inspiration, one that is complementary to more conventional methods. "It's important to go out and discover the facts for yourself," said Ronald Coase, who developed his ideas about the "nature of the firm" (1937) during a year of visits to firms throughout the US.

II. Many criticisms of fieldwork can be answered with improved methods

Many economists remain skeptical of qualitative research, fearing that it is not objective, replicable, or generalizable. "We don't have standards for what good fieldwork is," said Eli Berman. "In econometrics, we know to look for things like identification and specification issues, but what are the analogues in field work? How is it different from journalism?"

Better techniques can alleviate these problems. "Economists think that while econometrics requires years of training, field research is easy, " said Rebecca Henderson. "But we need to pay just as much attention to things like careful research design and sample selection as we do in quantitative research." Below are tips from economists and others experienced in field methods. (See Robert Yin (1989), Robert Thomas (1994) and www.nber.org/sloan for additional suggestions. David Lodge (1990) presents information in novel form.)

Objectivity. A way for economists to avoid confirmatory bias is to test hypotheses coming from several competing theories. (Economists often go into the field with hypotheses to test. In contrast, disciplines like anthropology emphasize understanding the world as their informants do.) One source of alternative theories is to let respondents talk, even when you think they are getting off the subject. You will probably learn something you never thought to ask about and besides, getting to tell their story is part of their payoff for talking with you (Michael Piore, 1979).

It is important to do more than look to confirm or disconfirm pre-conceived hypotheses. Claudia Goldin said about a 1995 visit to Joseph Pollak, an auto-parts manufacturer in Stoughton, MA: "I didn't have any particular expectations going into the plant, but I remember vividly looking down from a mezzanine from which you could see the whole shop floor. As I looked down, I realized I was observing--in one moment--the transition from 19^th century technology to 20^th century technology. I could see the relative increase in the demand for skill just scanning across the room. The 'continuous process' machinery required lots of skilled labor to set up the machines and mechanics to maintain them; there were few operators...The old-fashioned areas [making similar parts], however, were filled with semi-skilled workers and almost no skilled workers. The scene sparked my imagination and I wrote two papers (Goldin and Lawrence Katz, 1996, 1998). For years I had been reading the history of technology, but it wasn't until I went to Pollak that I made the connection that adoption of continuous-process technology was complementary to skill."

Another way to enhance objectivity is to "triangulate", asking questions so that answers can be checked against information from other interviewees, company documents, trade journals, etc. Site visits also provide a way to get a perspective other than the one presented by interviewees. A dramatic example of this came at our 1995 visit to LTV Steel in Cleveland, when workers stopped us to explain their anger at management's participation in a non-union joint venture in Alabama, which they thought violated the spirit of union-management partnership. Managers hadn't mentioned this dispute to us in their presentation on determinants of plant performance, even though it had caused the union to pull out of (productivity-enhancing) employee-involvement programs.

This incident also illustrates the value of interviewing people in a variety of positions. High-level people will give you an overview of the firm's intended strategy; low-level people offer detailed examples of the incentives and constraints they actually face.

Letting respondents tell their story does not mean taking everything they say at face value--you should be as skeptical of their statements (and as appreciative of them) as of any other data. Phrase questions concretely, and ask for examples. Until I asked for examples of trustworthy behavior in customer-supplier relations, I didn't realize that the definition affected the measured relationship between trust and performance. For example, one manager said, "I have a very trusting relationship with the plant I supply. If they find defects, they'll call us up and we can fix it, without anything ever showing up in the records." Another manager gave a quite different description of a trustworthy customer: "They are incredibly strict on quality--they'll send back a whole lot if even one part is defective. But they're always there to teach us, to make us better." The first manager felt little incentive to improve from his "trustworthy" customer; the second one felt a strong incentive.

Replicability. Often interviewees won't talk freely unless they are promised confidentiality, making it difficult for other researchers to replicate by interviewing the same people. However, there are other ways to enhance replicability. Rebecca Henderson and Iain Cockburn (1994) coded interview transcripts to create a dummy variable measuring whether pharmaceutical firms had "pro-publication" policies for their scientists. (This variable was a significant determinant of firm's research productivity. It is hard to imagine how else they could have obtained this information, other than by asking the actors directly about their incentives.) The more clearly described the coding, the more other researchers will be able to replicate it elsewhere.

Care in writing up results can increase replicability and readers' confidence in qualitative findings. For example, some field-research papers present arguments about what drives their regression results, and then say "interviews confirmed these findings", without providing information about how interviewees were selected, or enough information about what they said for readers to judge for themselves. This is like asking readers to believe a summary of econometric results without tables of regression coefficients. See David Levine (1993) on describing selection of interviewees, Stephen Barley (1986) on using quotes as evidence, and Kathleen Eisenhardt (1989) on using 'textual tables' to present this information concisely.

Some changes at journals could alleviate the space problems that cause economists to truncate description of their fieldwork. Authors could use journal web sites to post information about field findings that didn't fit in the published article, and more journals could accept stand-alone qualitative analyses (the Journal of Financial Economics, Journal of Industrial Economics, and Industrial Relations already do).

Generalizability. How can one be sure that one's conclusions go beyond the few firms one has visited? Careful selection of cases according to theoretical principles, and use of cross-case analysis can alleviate this problem (Eisenhardt, 1989). If one is building theory, in-depth understanding of a few cases may be appropriate. If one wants to use observations from plants as data, visiting a large number of them is quite convincing (Ichniowski, et.al, 1997). As more cases are done on a topic, the more we will understand what is specific to each setting, as the piece-rate example shows. That is, the solution to the generalizability problem is to do more field research, not less!

No methodology is perfect. "Regressions also have serious problems of generalizability (they predict poorly out of sample), subjectivity (researchers may stop specification searches when their favorite t statistic rises over 2), and measurement error (critical concepts like 'income' and 'capital' are very poorly measured). This is why research is hard--and why we should believe only findings obtained with multiple methodologies," said Levine.

In summary, good field research should be like good journalism in containing accurate, vivid examples. It should go beyond journalism in explaining potential biases in selection of cases, describing the construction of concepts used, and having a theoretical starting point and/or outcome. In conclusion, field research can make us better economists--whatever our current technique--by increasing our understanding of the objectives, constraints, and incentives that economic actors face.

REFERENCES

Berra, Yogi. "Famous Yogi Berra Quotes." http://www.yogiberraclassic.org/quotes.htm, nd.

Barley, Stephen. 'Technology as an Occasion for Structuring: Evidence from Observations of CT Scanners. " Administrative Science Quarterly, 1986, 31, pp. 78-108.

Coase, Ronald. "The Nature of the Firm." Economica, 1937, 4 pp. 386-405.

Eisenhardt, Kathy. "Building Theories from Case Study Research." Academy of Management Review, 1989, 14(4) pp. 532-550.

Freeman, Richard and Kleiner, Morris. "The Last American Shoe Manufacturers." NBER Working Paper 6750, 1998.

Goldin, Claudia and Katz, Lawrence. "Technology, Skill, and the Wage Structure." American Economic Review, May 1996, 96(2) pp. 252-257.

_______. 'The Origins of Technology-Skill Complementarity." Quarterly Journal of Economics, August 1998, 113 pp. 693-732.

Susan Helper. "Strategy and Irreversibility in Supplier Relations." Business History Review, 65(4), Winter 1991, pp. 781-824.

_______."Complementarity and Cost Reduction." NBER Working Paper 6033 (revised), 1999.

Henderson, Rebecca, and Cockburn, Ian. "Measuring Competence? Exploring Firm Effects in Pharmaceutical Research," Strategic Management Journal, 1994, 15, pp. 63-84.

Ichniowski, Casey, Prennushi, Giovanna, and Shaw, Kathryn. "The Effects of Human Resource Management Practices on Productivity." American Economic Review, May 1997, 87(3) pp. 291-313.

Landers, Renee, James Rebitzer and Lowell Taylor. "Rat Race Redux: Adverse Selection in the Determination of Work Hours in Law Firms." American Economic Review, June 1996, 86 (2), pp. 329-348.

Lazear, Edward, "Performance Pay and Productivity," NBER Working Paper 5672, 1996.

Levine, David I. "Fairness, Markets, and Ability to Pay: Evidence from Compensation Executives." American Economic Review, December 1993, 93(4),pp. 1241-1259.

Lodge, David Nice Work. New York: Penguin, 1990.

Piore, Michael. "Qualitative Research Techniques in Economics." Administrative Science Quarterly, December 1979, 24, pp. 560-569.

Smith, Adam. An Inquiry into the Nature and Causes of the Wealth of Nations. New York: Modern Library, 1985.

Thomas, Robert. What Machines Can't Do. Berkeley: University of California Press, 1994.

Yin, Robert. Case Study Research. Beverly Hills: Sage Publications, 1984.

Zucker, Lynne, Darby, Michael, Brewer, M. "Intellectual human capital and the birth of US biotechnology enterprises." American Economic Review March 1998, 88(1) pp.290-306.

^1. Economics Department, Case Western Reserve University, Cleveland OH, 44106. I thank Martin Feldstein and Adam Jaffe for involving me in the NBER/Sloan program, and David Levine for extensive discussions. I also received much useful advice from those cited herein, Zvi Griliches, Carol Heim, and Don Siegel, The quote is from Yogi Berra (nd).