This project could not have been completed without generous contributions of time for discussions by individuals at NASA and a number of private firms. We thank, without implicating, Daniel J. Adams, Tim Gurin, Richard Hullihen, Sylvia Kraemer, Norman Smith, Allan Vogele, and Warren W. Wolf. We also benefited from comments at the NBER Productivity Lunch, and from the able research assistance of Margaret Lister Fernando and Chu Chi-Leung. We are grateful for financial support from the Alfred P. Sloan Foundation, via the National Bureau of Economic Research Project on Industrial Technology and Productivity. Bruce A. Bank’s participation in this research project was as an individual and not an employee of NASA-Lewis.

2. Introduction

Federal research institutions comprise a significant component of the U.S. research infrastructure. The approximately 700 Federal labs are extremely heterogeneous, varying from the large "National" laboratories of the Department of Energy such as Las Alamos and Oak Ridge, to small highly specialized facilities. They include "intramural" facilities that are owned and operated by the Federal government, and also Federally Funded Research and Development Centers ("FFRDCs"), which are operated by a university, a private firm or a non-profit organization but receive all or most of their funding from the Federal Government. Examples of intramural labs include the National Institutes of Health, the National Institute of Standards and Technology (NIST) within the Department of Commerce, and the research centers of the National Aeronautics and Space Administration (NASA). Examples of FFRDCs include the DOE National Labs and the NASA-funded Jet Propulsion Laboratory operated by Cal Tech. In 1995, approximately $25 billion of R&D was performed at federal research institutions, which is about 14 percent of the aggregate U.S. research effort, and about 41 percent of Federal spending on R&D. For comparison, universities—the other major locus of public research—performed about $22 billion of R&D, of which about $13 billion was funded by the Federal government, only 21% of Federal research expenditures (National Science Board, 1996).

The last 15 years have seen increasing policy and academic interest in the role of Federal labs in commercial innovation. Beginning with the Stevenson-Wydler Technology Innovation Act in 1980, and continuing with the Federal Technology Transfer Act of 1986, the National Competitiveness Technology Transfer Act of 1989, and the Defense Conversion, Reinvestment, and Transition Assistance Act of 1992, Congress has implemented statutory changes explicitly designed to foster the transfer of technology from the public sector into the private sector. The biannual Science and Engineering Indicators now publishes a wealth of statistics documenting technology transfer activities at the labs. There have been several studies by the Congressional General Accounting Office of such activities.

This paper analyzes patents awarded to the U.S. government, and the citations received by those patents from subsequent patents taken out primarily by private firms. The citation analysis allows us to examine two aspects of the production of commercially relevant technology by the federal government. First, we use the frequency and diversity of citations received to measure the "importance" and "generality" of the inventions that are emanating from federal laboratories. We examine how these have changed over time. Second, we examine the geographic locus of citing patents to infer where the subsequent commercial development of technologies that originate at the labs takes place. We focus particularly on patents awarded to NASA, but also examine patents awarded to other federal agencies.

There is a large and growing literature on the use of patents and patent citations to infer the technological output and impact of different institutions. As discussed further below, there is evidence confirming the validity of patent citations as a measure of technological impact. There is not, however, a clear understanding of the exact nature of the relationship between citations and technology flows. For this reason, in addition to the quantitative analysis, we undertake a detailed qualitative analysis of the citations to a small number of NASA patents in order to gain a richer understanding of the inferences that can be drawn about technological impact and its geographic distribution by examining citations data.

3. Patents and Patent Citations
1. Background

There is a long history in economics of the use of patent data to understand the processes of invention and innovation (Schmookler, 1966; Griliches, 1984 and 1990; Jaffe 1986). A patent is a temporary monopoly awarded to inventors for the commercial use of an invention. For a patent to be granted, the invention must be non-trivial, meaning that it would not appear obvious to a skilled practitioner of the relevant technology, and it must be useful, meaning that it has potential commercial value. If a patent is granted, an extensive public document is created, containing detailed information about the invention, the inventor, the organization (if any) to which the inventor assigns the patent property right, and the technological antecedents of the invention. All of this information can be accessed in machine-readable form. As a result, there has been much recent interest in exploiting this more detailed information to understand the process of invention.

In particular, the "references" or "citations" that appear in a patent identify earlier inventions whose claims are sufficiently close to the claims of the citing patent that the patent examiner deems it necessary to identify them. The citations that appear in a patent serve the important legal function of delimiting the property right granted by the patent, by identifying "prior art" that is not covered by the property right granted in the citing patent. Thus the citations contained within a patent convey information about the technological antecedents of the invention covered by the patent. Conversely, we can identify the subsequent patents that later make citations to a given patent, thereby learning something about the technological descendants of the cited invention.

Previous research has used information on citations made and received by patents to draw inferences about the process of technological accumulation (Caballero and Jaffe, 1993); about the "basicness" or importance of particular inventions (Trajtenberg, Henderson and Jaffe, forthcoming); about changing patterns of importance of university patents over recent decades (Henderson, Jaffe and Trajtenberg, forthcoming) and about the extent to which knowledge spillovers are geographically localized (Jaffe, Trajtenberg and Henderson, 1993). Trajtenberg (1990a and 1990b) has shown that there is a correlation within a family of products between the number of citations received by the patent covering a particular model and the consumers’ surplus generated by that model.

In this paper, we use information about the citations made to federal patents in two ways. First, we compare the total number of citations received and the technological diversity of those citations to compare the average technological impact of NASA patents, other federal patents, and a random sample of all patents. Second, we look at the geographic location of the inventors who cite NASA and other federal patents, to see the extent to which subsequent commercial development of ideas related to federal research occurs nearby, elsewhere in the U.S., or overseas.

2. Details

The data are drawn from the universe of patents granted by the U.S. patent office between 1963 and 1994, with information on the citations made to these patents for the period 1977-1994. The computerized patent office files only identify the "assignee" by name beginning in 1969, although the data back to 1963 are coded to distinguish patents on the basis of whether the patent was assigned to a corporation, the government, or unassigned. For this reason, we can only identify patents assigned to NASA beginning in 1969, but we can identify U.S. government patents throughout the 1963-1994 period. We construct three distinct sets of patents for analysis. The first consists of the 3782 patents assigned to NASA between 1969 and 1993 (the "NASA sample") The second sample consists of the 37939 patents assigned to the U.S. government between 1963 and 1993, excluding those identified as belonging to NASA (the "other federal" sample). This sample includes patents from 1963-1968 assigned to NASA. For the period after 1969 when we are able to identify the specific department or agency to which the assignment was made, 67% are assigned to the armed services, 19% are assigned to the Department of Energy, 6% are assigned to the Department of Agriculture, and 3% are assigned to Health and Human Services (which includes the National Institutes of Health).

The last sample consists of a 1 in 100 random sample of all patents granted to inventors residing in the U.S. (the "random sample"). This sample contains 13997 patents for the period 1963-1993. Thus pooling the NASA, other Federal and random samples yields the basic dataset used in the analysis, which contains 55710 patents.

Construction of "Importance." We constructed the total number of citations made to each of the 55710 patents in the basic sample from patents granted between 1977 and 1994. (Note that one patent received 103 citations.) We call this citation count "importance." As can be seen from Table 1, the average patent in the dataset received 3 or 4 citations from patents between 1977 and 1993. Obviously, importance as we measure it differs greatly across time cohorts. Very recent patents will have received very few citations by the end of the period, and very old patents will have received many of their citations before 1977. Hence importance can only be compared across patents with the same grant date. In addition, citation practices may vary somewhat across different fields of technology. For descriptive purposes, we will categorize patents into 5 broad fields. For statistical analysis, we control for technological field effects using "dummy" variables for each 3-digit patent class used by the patent office to classify technologies. There are approximately 400 such classes spanning the range of patentable technologies.

Construction of Generality. Following Trajtenberg, Henderson and Jaffe (forthcoming) we calculated the "generality" of each patent by examining the distribution of citations received by the patent across the 3-digit patent classes. Specifically, we took the fraction of citations received from patents classified into each patent class, and summed these squared fractions to yield a Herfindahl-like index of the concentration of citations across patent classes. We define generality as one minus this Herfindahl index of concentration. Thus generality is an index that is zero if all citations are received from patents in the same class (i.e., the Herfindahl concentration index is unity), and which approaches unity for patents whose citations are extremely dispersed. As shown in Table 1, average generality is about .3; there are many patents with generality of zero, and the maximum generality in our sample is .92. Note that generality can only be computed for the 40865 patents that received at least 1 citation.

Geographic Analysis. For the geographic analysis, we limit ourselves to patents coming from 5 "states" that have significant NASA facilities: California, Ohio, Texas, Virginia/DC/Maryland, and Alabama (the "NASA complex"). This analysis is also limited to those patents receiving at least 1 citation. This sample is 19776 patents. For each of these patents, we calculate the fraction of citations coming from the same "state;" the fraction that comes from any of the "NASA complex" states other than the state of the cited patent; the fraction coming from states other than NASA complex states; and the fraction coming from the rest of the world.

4. NASA and Other Federal Patents: Trends and Patterns
1. Totals over time

Figures 1A-1C show the trends in 3 patent series: NASA, other federal, and aggregate patenting over time, broken down by 5 major technology fields. Not surprisingly, the rate of NASA patenting at the end of the 1960s was much higher than in subsequent years. From a peak of almost 350 patents granted in 1971, patenting fell rapidly to a low of about 75 patents per year between 1979 and 1982. This increased to about 150 per year in the mid 1980s, fell slightly in the late 1980s, and has recently recovered to the level of 150 per year.

The interpretation of the NASA patent series over time is clouded by changing NASA policy with respect to patenting by NASA contractors. By law, NASA retains the property right in any invention developed by private firms using NASA research or procurement funds. Such firms can, however, seek a waiver of this right in order to patent commercially valuable inventions in their own name. If such a waiver is granted, then any patent that results will be assigned to the firm and will not show up as a NASA patent in our data. On the other hand, NASA has the right to deny the waiver and seek a patent in its own name, in which case our data would include a "NASA" patent that was funded by NASA money but which did not come out of research performed at a NASA facility. In principle, therefore, our NASA sample is a mixture of unknown proportions between patents from research performed at NASA facilities, and patents from NASA contracts or grant research where waivers were denied. We were told that the waiver policy became increasingly lenient through the 1970s. This resulted in a situation by the early 1980s where waivers were essentially automatically granted. All else equal, this would produce some decline in the number of patents in our NASA sample, as patents from contract research where waivers were denied gradually disappear.

"Other Federal" patents remained approximately flat at about 1600 per year from 1965 through 1976, and then fell significantly, to the rate of about 1000 per year, which has held until today, with the exception of a dip in the late 1980s. Unlike NASA, the Defense Department, the Department of Energy and DOE’s predecessor, the Atomic Energy Commission, do not require waivers before their contractors can patent inventions derived from funded research. This means that the "other Federal" sample is not likely to be significantly contaminated by patents that do not originate at the labs. On the other hand, it also means that many patents originating at FFRDCs (including the large National Labs) may appear in the name of the operator of the facility rather than the government. This difference has to be considered when comparing NASA patents to other Federal patents.

The historical pattern of total patents granted per year, as indicated by our random sample, is similar to that for federal labs other than NASA: a plateau through the late 1970s, followed by a decline and then leveling off. Note that all three series show an increase in patenting some time around the late 1980s. However, the exact timing differs somewhat in each case.

The rate of patenting is determined by the scale of research effort, the success of that effort, and the "propensity to patent" whatever research outputs have been produced. If the statutory activity regarding technology transfer has affected the behavior of the labs, we would expect that the labs’ propensity to patent was increased. Assuming that the success rate has not changed systematically over time, we can get an indication of changes in the propensity to patent by examining the ratio of patents to R&D effort, proxied by expenditure. Figure 2 shows patents per million dollars of R&D expenditure for the three samples over time. In order to make broad movements more visible, and allow for the lag of 1 to 3 years between patent application and patent grant, each point in Figure 2 takes a moving average of lab research expenditures and divides that research number into a moving average of the number of patents, moved forward 2 years. Figure 2 shows that the propensity to patent for all domestic patents declined steadily up until about 1981. It has been roughly constant since. For non-NASA federal patents, the decline continues until about 1986. Then there has been steady growth. Though the rise in the propensity to patent since 1986 is consistent with the policy mandate for increased commercial relevance, it is worth noting that by the early 1990s labs’ propensity to patent had only returned to what it was in 1978.

Figure 2 shows that the steep decline in NASA patenting shown in Figure 1A over the 1970s was a combination of declining NASA research spending and declining propensity to patent at a rate approximately comparable to that for both other federal and all domestic patents. Throughout this period, NASA’s propensity to patent is somewhat below that for other federal labs. In the early 1980s there was a significant increase in NASA patents per dollar of research expenditure, up to approximately the level of the other federal labs. Since then, NASA patenting has moved in approximate proportion to research expenditures, leaving the ratio approximately constant.

2. Importance and generality

Figure 3 shows some comparisons of the 3 groups across technological fields, averaged over the entire time period. For the random sample of all domestic patents, 35% are mechanical; 25% chemical (excluding drugs); 19% relate to electronic, optical or nuclear technology; 6% are for drugs or medical technology; and 15% are in the residual "all other" category. The NASA patents exhibit a share for "electronics/nuclear" that is double that of the universe. There are offsetting reductions in all other categories, particularly chemicals and "all other." Other federal patents have a concentration in the electronics/nuclear category that is similar to NASA’s, but have a much lower share in the mechanical category than either NASA or the universe.

The lower 2 panels of Figure 3 compare the importance and generality of the three groups of patents within each of the five fields. It should be noted that, unlike the regression results described below, these comparisons do not control for time effects or for differences across technologies within the five broad fields. Still, the patterns are interesting. NASA patents are the most important, on average, in 3 of the 5 fields. They are the most general in all 5 fields. In contrast, other federal patents are, on average, the least important of these 3 groups in all 5 fields. The differences are moderately large. Other Federal patents do better when measured by generality; although they lag NASA patents, they are close to as good or better than the random sample in every field except "all other." We will see below that some of the differences shown in Figure 3 remain after controlling for time and more detailed technology area effects, while some do not.

Figure 4 shows the relative importance and generality of the different samples over time (ignoring differences across fields). The rising and then falling pattern in all series is an artifact of the data construction. As noted above, both very recent and very old patents tend systematically to have fewer citations in our sample For this reason, we will not attempt to draw any conclusions about changes in the importance or generality of the random sample over time. We will focus on these measures for the two federal samples relative to the random sample. Figure 4 suggests that non-NASA federal patents are systematically less important than the random sample, and were also less general up until about 1975. In contrast, NASA patents are comparable in importance to the random sample. There was a period in the late 1970s when they appear to have been more important on average. Their generality is also comparable. Again, there was a period in the late 1970s, perhaps extending into the early 1980s, when they became more general. In the next section, we will test the robustness of these findings to controlling for technology field effects.

3. Geographic impact

Figure 5 shows the fraction of all patents containing citations to NASA patents, by state of the citing patent. Five "states" (Virginia/Maryland/DC, New Mexico, Alabama, Washington and California) have 2 percent or more of their patents containing citations to NASA patents. For perspective, NASA patents in the aggregate comprise less than 2-tenths of one percent of all patents. An additional four states (Arizona, Massachusetts, Ohio and Utah) have NASA citations in more than 1.5% of their patents. It is clear from Figure 5 that no one factor explains high citation rates to NASA patents. States like New Mexico and Washington do not have NASA facilities, but do have research enterprises that are focused on technological areas where NASA plays an important role. Virginia/Maryland/D.C., Alabama, Ohio and California all have NASA facilities and high rates of NASA citation. The only state with a significant NASA research presence that does not have a high citation intensity is Texas, which ranks 17th at 1.2%. This probably reflects the shear size and diversity of the state in which pockets of NASA-related activity are not of sufficient size relative to other patenting areas (such as petrochemicals) to affect the overall NASA citation intensity.

Figure 6 exhibits the fraction of all patents in each state that contain citations to the "other Federal" group of patents. This distribution is considerably more skewed. Whereas federal patents comprise 2-3 percent of all patents over this time period, almost half of all patents in New Mexico contain citations to federal patents. This reflects the extreme dependence of that state’s research activity on the federal government. Other states with very high fractions are Alabama and Virginia/Maryland/DC. Following these are Washington, Utah and California, all states with significant federal lab presence. As with citations to NASA patents, the presence of major federal facilities is one explanation for high citation intensities, but it clearly is not the only factor. The regression analyses below test its significance.

5. Regression analysis of citations

Table 1 presents simple statistics for these 3 samples. Based on Figure 3, it should not be surprising that NASA patents are more important and more general than the random sample, while the other Federal patents are less important and slightly less general. Note, however, that the average vintage differs across the groups. NASA patents are on average younger (because we have none from 1963-68) and the federal patents on average older (because federal patenting in recent years has grown more slowly than overall patenting). Because it is invalid to compare citation intensities of different vintages, these differences must be controlled for in a rigorous comparison of the groups.

1. Importance and generality

Because of the large number of data points in our samples, it is possible to incorporate a large number of control variables in the analysis. Our basic strategy is to include dummies for patent class and grant year of the cited patent in all analyses. For regressions involving the entire dataset, this involves 30 "year dummies" and approximately 390 patent class dummies.

The simplest analysis merely regresses importance and generality on the year and class dummies, plus dummies for being from the NASA or "other Federal" subsamples. These dummy coefficients thus reflect the average difference between the NASA and other federal subsamples, relative to the random sample, controlling for patent class and cohort effects. Motivated in part by Figure 3, this analysis was carried out separately for each of the five broad technical fields. The results are presented in Table 2. Comparison of Table 2 and Figure 3 indicates that the simple calculation of means in Figure 3 can be somewhat misleading. With respect to importance, NASA patents are significantly more important than the random sample in the mechanical category, and significantly less important in the chemicals and electronics/nuclear category. Consistent with Figure 3, the Other Federal patents are significantly less important than the random sample in all five categories. With respect to generality, NASA mechanical and all-other patents are significantly more general than the random sample and are statistically indistinguishable from the random sample in the other categories. The other Federal patents are statistically indistinguishable from the random sample in all five categories.

Turning to the patterns over time, Table 3 and Figures 7 and 8 present the results of regressions in which the NASA and other Federal dummies are interacted with year dummies. This allows us to estimate a separate difference between these groups and the random sample for each year. We continue to control for patent class effects, although we do not allow the differences among the groups to vary by technology field. In this case, the results, after controlling for other effects, broadly confirm the simple comparisons shown in Figure 4. Non-NASA federal patents are less important than the random sample throughout the period, although this difference is not statistically significant by the 1990s. In contrast, NASA patents appear to have had a "golden age" in the second half of the 1970s in which their importance exceeded that of the random sample, followed by a decline through the 1980s, so that they are indistinguishable from other federal patents by the second half of the 1980s.

Because the distribution of citations across patents is extremely skewed, it is worth asking whether the apparent "golden age" in NASA patenting might be driven by a small number of extremely highly cited patents. Looking at the complete citation distribution by year for the NASA patents suggests that this was not the case. In particular, there is no identifiable change over time in the ratio of the mean number of citations to the median. This suggests that the higher citation rate during this period was not confined to a handful of patents.

With respect to generality, non-NASA federal patents are less general than the random sample in the 1960s. However, they improve their relative importance to essentially par from the late 1970s through the present. NASA patents are more general than the random sample from about 1975 through about 1985, with NASA relative generality showing a decline that mirrors that for importance. However, the superiority in generality lasts slightly longer.

An interesting question that we cannot answer is whether the disappearance over the 1970s of the contractor patents for which waivers were denied relates to the changing importance and generality of the NASA patents. If this change alone were to explain the patterns in Figures 7 and 8, what would have to be true is that the contractor patents that were assigned to NASA before waivers became automatic were systematically more important and more general than other NASA patents. There is no apparent reason why this should be true. Technological significance was never a criterion for granting or denial of a waiver; before waivers became automatic, they were denied in cases where the case for the commercial value of the invention was weak. Still, the close coincidence of the timing of disappearance of the contractor patents and the decline in average importance suggests there may be a relationship.

2. Geographic Localization

In order to investigate the geographic localization of the patent citations consistently across our data, we focus our attention on a subset of each of our three subsamples -- those patents whose primary inventor resides in one of the states from which a significant number of NASA patents originate: California, Ohio, Texas, Alabama, Maryland, Virginia and the District of Columbia. We refer to this set of states as the "NASA complex." We include in our analysis all citations to these patents, regardless of where they come from. As shown in the top part of Table 4, both the NASA patents and the other federal patents get fewer of their citations from the same state as the cited patent in comparison to the random sample. Almost all of this difference is then made up in a commensurably higher rate of citation from the other NASA complex states; so the fraction of citations from other U.S. states and other countries are similar across the groups. Of course, these contrasts do not control for differences in either technological field or timing across the groups.

To control for such differences, we regressed the fractions on dummies for cited year, dummies for patent classes, and dummies for NASA and other Federal. In addition, for this component of the analysis we also included dummy variables for the specific states. What this means is that the coefficients on NASA and "other Federal" are not influenced by the fact that more or less of the cited patents in those groups come from particular states. The results are presented in the bottom half of Table 4. Controlling for time, state, and patent class effects reduces but does not eliminate the differences among the groups. NASA and other federal patents are significantly less likely to receive their citations from the same state, and significantly more likely to receive them from the other NASA complex states. For the NASA patents, we interpret this to mean that the relevant "local" area is, to a significant degree, the complex of states where NASA operates. This interpretation is strengthened by the fact that the fraction of citations from other U.S. states and from foreign countries is indistinguishable from the random sample.

Other federal patents show a pattern similar to NASA, though somewhat weaker. We interpret this to suggest that there is probably a larger, somewhat more complex "complex" for other federal labs, but it overlaps significantly with the NASA complex as defined here. This issue obviously requires further study.

The localization effects shown in Table 4 are influenced by "self-citations." We define these as citations coming from patents assigned to the same organization as the cited patent. Self-citations are more likely to be geographically localized than other cites. In a world of global organizations they are by no means exclusively local. For some purposes, such as examining the role of new technology in regional economic growth, the fact that a disproportionate fraction of local citations may be self-cites may not be an important consideration. If, however, we desire to use citations as indicators of possible paths for research spillovers across organizations, it makes sense to exclude self-citations. Further, since the extent of self-citation likely differs between government and private research organizations, it is useful to know whether comparisons between the groups are affected by the presence of self-citations.

Table 5 reproduces all of the results of Table 4, excluding all citations that are self-citations. Hence the fractions from the same state, NASA complex, etc., are calculated excluding self-citations in both the numerator and denominator of the fractions. Comparing Tables 4 and 5 reveals that self-citations are indeed more localized. In the raw data, self-citations come from the same state 45 to 60% of the time, depending on the sample. Excluding self-citations (which comprise about 10% of citations for the federal labs and about 15% for the random sample) thus lowers the fraction from the same state by roughly a third. Interestingly, the fraction of citations from other NASA complex states is virtually the same in Tables 4 and 5. This implies that the fraction of self-citations from those states is about the same as the fraction of non-self-citations. The decline in the same-state shares going from Table 4 to Table 5 is offset primarily by increases in the foreign and "all other U.S." shares.

The bottom half of Table 5 examines localization fractions for non-self-citations controlling for patent class and state effects. The results are similar to those in the bottom half of Table 4, except that the lesser same-state localization for NASA and to some extent "other Federal" is somewhat attenuated when self-citations are excluded. What this says is that part of the less localization of the federal lab patents as compared to the random sample in Table 4 results from the fact that citations to non-federal patents are more likely to be self-citations; also, self-citations are more likely to be localized. Once self-citations are excluded, citations to NASA patents are about 2 percentage points less likely to come from the same state. Since the raw same-state fraction (excluding self-citations) for the random sample is about 13%, this is a non-trivial difference.

This lower same-state fraction is almost entirely offset by a higher fraction of citations in the other NASA complex states. Consequently, the fraction coming from the rest of the U.S. and from abroad is not distinguishable from the random sample. In other words, the fraction of citations coming from the entire NASA complex (including the same state) is about the same for all three groups of patents. Within the NASA complex, however, NASA citations, and, to a lesser extent, other federal lab patents, are less concentrated in the state of the cited patent.

 

6. Qualitative Evidence on NASA Patents and Patent Citations

Our analysis of NASA’s patents provides some intriguing evidence on technology spillovers from federal labs. The analysis rests, however, on the explicit assumption that patent citations reflect flows of knowledge, and that citations received tell us something about the technological significance of patents and the underlying inventions. In this section we explore qualitatively the relationships among patents, patent citations and technology spillovers. We have three main purposes for this part of the paper: (1) Use the qualitative information gained from interviews to improve quantitative analyses of federal lab commercialization using patent data; (2) Develop a deeper understanding of the pathways through which federal lab research influences firm performance (in this case, innovation by firms); and (3) Validate the use of patent citations for analyzing knowledge flows.

To achieve these objectives, we undertook a detailed analysis of the invention and patenting of the Electro-Physics Branch (EPB) of the NASA-Lewis Research Center located in Cleveland, Ohio, which is directed by one of the authors. In addition to Banks’ intimate knowledge of the inventions, patents, and technology transfer activities of the EPB, we also talked with a number of other people with knowledge about patenting and technology transfer involving government labs. EPB is involved in the development of high performance durable power materials and surfaces technology to meet NASA, national, and U.S. industrial needs.

1. Evaluating EPB’s Patents and Patent Cites

We began by identifying EPB’s thirty-eight patents as of October 1996. These patents had received 160 citations through 1994, of which 139 are from other organizations. Seven EPB patents received more than ten citations, representing nearly 70% of all citations. One stood out with 33 citations (#4,490,229: "Deposition of Diamondlike Carbon Films"). Two of EPB’s patents have been licensed: patent # 4,620,898 has been licensed to J.P. Technologies (long-life sputter masks) and # 4,560,577 which is licensed to Air Products and then assigned to Diamonex, Inc. (dual beam deposition of diamond-like carbon films).

We analyzed in detail citations made to the seven EPB patents that received 10 or more citations overall. We limited the analysis to citing patents that themselves received 10 or more citations ("important" citing patents). This resulted in 26 citing patents. The relationship between these 26 citing patents and the EPB patent that they cite was analyzed along three dimensions. First, Banks determined whether the inventors or the organization associated with the citing patent had ever had personal communication contact with EPB. This characterization was made based on Banks’ lab notebook records. To further explore the relationship between citations and communication, we identified 18 additional patents which were assigned to organizations known by Banks to have had extensive contact with EPB, and whose subject area was closely related to those of EPB patents.

Second, in order to get a sense for the extent to which citations truly correspond to our concept of inventions building upon one another, Banks characterized the technological relationship between the inventions captured in the citing and cited patents. This characterization took the form of determining whether the technologies embedded in the two patents were the same, related or unrelated; and whether the application to which the technology was put was the same, related or unrelated. In other words, each citing/cited patent pair was placed in a 3X3 matrix indicating whether the technology and its application were the same, related or unrelated.

Finally, irrespective of communication or lack thereof, Banks characterized the likelihood that each invention described in a citing patent had benefited from any kind of knowledge spillover from the cited invention. He categorized each cite on a scale of 1 through 5 (1 =spillover highly unlikely and 5 =spillover certain). This was done on the basis of such evidence as parallel citations to related scientific papers and details of the invention that seem to indicate influence.

Figure 9 presents the data on each of the 26 EPB patent citations plus the 18 patents not containing citations. The 2x2 matrix groups each citation based on whether there is evidence of a direct contact with EPB (yes, no) and whether there is a citation to an EPB patent (yes, no). The number in parentheses refers to the number of citing/cited pairs of each type. The letters A through I correspond to the nine combinations: same technology/same application, same technology with related application, etc. The number next to the A through I designation represents the spillover rank. So, for example, in the "no contact/ yes cite" cell, (3) A4 means there were 3 A cases (same process or technology and same application) with strong evidence of technology spillover (a rank of 4 on the scale of 1 through 5).



Technological relationship between inventions connected by citation. These qualitative results indicate considerable diversity of technological connections underlying citations. First, approximately one-third (8/26) of citations are cases where neither the technology nor the application are clearly related to the cited patent (category I). The source of these apparently spurious citations is not entirely clear, although some views on their origin from industry participants is discussed below. Second, at the other extreme, there are a handful of citations (3/26 or nearly 12%) where both the technology and the application are essentially the same as the cited patent (category A). In theory, an invention that uses an existing technology for an existing application should not be patentable; these citations appear to represent the margin of patentability where minor variations in approach or features suffice to achieve the novelty or "non-obviousness" required for patenting.

The remainder of cases (15/26 or or roughly three-fifths) correspond to situations in which citing patents utilize the same or related technologies in unrelated applications; or conversely, apply unrelated technologies in similar applications. We believe these partially related inventions (which comprise the majority of all citations) correspond reasonably well to our conception of citations as indicating some sort of cumulative technological impact.

Evidence of influence or knowledge spillover from cited to citing inventions. In order to understand the extent to which citations can be used as a proxy for flows of knowledge between inventors, Banks examined evidence of explicit communication between EPB scientists and the inventors on patents that cite EPB patents. In addition, Banks provided a list of firms that he knew had learned from and utilized EPB research, and a systematic search was made to determine if the patents of such known "spillover recipients" contained evidence of that spillover in the form of citation to EPB patents. In other words, we looked for both "Type 1" error (citations occur but spillovers do not) and "Type 2" error (spillover occurs without citation) in the identification of citations with knowledge spillovers.

It is clear from Figure 9 that citation occurs without there having been known, direct communication; only 4/26 citations (15%) involved known contact. If, however, we take "3" or better as evidence of spillover, then 58% (15/26) of cites appear to involve technology spillovers. This suggests that most spillovers result from indirect rather than direct communication, such as the citing inventor reading papers or patent documents, or even attending a presentation by the cited inventor without identifying herself. This characterization would seem to be consistent with the econometric finding that geographic proximity does increase the likelihood of citation, but only modestly.

It is also clear that contact can occur and not generate any citations: 18 patents were found in the general area of EPB’s research by organizations that had had significant contact with EPB but that did not cite the EPB patents. It is interesting, however, that in only 2 of the 18 patents where contact occurred but there was no citation does Banks conclude the evidence of spillover is fairly strong; in the other cases the patented inventions are not directly related to EPB’s, and there is little evidence of spillover.

Suffice it to say that citations are clearly a noisy indicator of spillovers. The overwhelming majority of citations do not appear to have involved known contact with EPB personnel. On the other hand, in many cases there is indirect evidence that the citing inventors knew about and apparently benefited from EPB research. Such evidence includes citations to scientific papers authored by EPB or related people, or details of the invention specification that strongly suggest prior knowledge of the EPB invention. Conversely, contact does not always lead to citation. In most cases the lack of citation appears to result from a lack of direct spillover benefit from the contact, at least to date, but in a few cases apparent spillovers occurred despite the known communication and the absence of citation.

2. Insights from participants in the patenting and technology transfer processes

Our interviews with EPB and various companies point to a lot going on below the surface of patent citations. In interpreting patent citations, we have talked with three kinds of participants in the patenting process: the inventor, the R&D director, and the patent attorney. While each participant contributes important information about citations, our interviews have made it quite clear that it is the inventor who knows whose technology has been most important in the invention and the pathways by which the knowledge traveled (consulting, reading of papers, hiring of graduates, etc.).

At the broadest level, our discussions with practitioners confirm that quantity of citations to a patent is a valid indicator of an invention’s importance. While there will be exceptions in individual cases, it is clear that there is a systematic relationship between the depth and breadth of an invention’s technological impact and the number and diversity of citations it receives. However, participants in the process are less convinced that citations are a strong proxy for the actual path of knowledge flow (i.e., if A cites B, it doesn’t always mean that A learns directly from B.). Because many parties potentially play a role in the decision to include a citation, it is difficult to get a clear picture of the likelihood that citations occur in cases where the inventor had no knowledge of the technology underlying the cited patent.

It is clear that citations don’t always represent what we typically think of as knowledge flow. Some of what we see represents noise. The reason is that citations get included for various reasons, which can be several types: cites involving recent knowledge flow (the inventor actually learned something important from another researcher either directly or indirectly); strictly legal cites (the patent attorney clouds the picture by acting in a risk-averse manner to avoid any hint of infringement, opting to include a citation even when the inventor doesn’t consider it as prior art); after-the-fact cites (knowledge of a related technology is gained after the invention); teaching cites (patents that everyone considers basic, even if they are old); and cites added by the patent examiner. Unfortunately, at present we have no way to determine the frequencies attached to each category, although it appears from interviews that knowledge content increases more than in proportion to the quantity of citations as the number of cites increases.

On the decision to include a cite or not, the attorney will typically win out – in other words, opt to include the cite to be safe. (Interviews with patent attorneys at NASA and with private companies indicate the seriousness of the disclosure process. For instance, a common practice is to require the inventor to sign a form signifying that all known state-of-the-art work has been cited, including patents and non-patent sources.)

Not surprisingly, there is general agreement that patent citations are more indicative of patterns of knowledge flow at the level of organizations than at the level of individual patents. A number of organizations have begun to use patent citation information to identify centers of related technological activity around the world. Other firms who have not investigated citation patterns were clearly captivated when we showed them who was citing their patents. Typically, they quickly recognized most citing organizations’ involvement in the technologies, particularly if the citing party is toward the top of the citing patents list based on number of cites to the cited patents (i.e., organizations responsible for a high percentage of citations to their patents). What seems true is that organizations frequently citing a firm or federal lab are typically well known by the cited party, acknowledging their mutual technological dependence. Also, companies appear to be far more attuned to citing assignee activity than either government labs or universities. Of course, this shouldn’t be surprising given their focus on profit and strategic advantage.

Interviews indicate that some firms have begun to pay increasing attention to external sources of technology and are more closely scrutinizing the technology capabilities of federal labs. It’s clear from an examination of the patent files that a fairly prominent set of large R&D-performing companies appear to have technology that is closely connected to federal lab technology. (This is shown by there being a relatively short list of firms responsible for the lion’s share of patent citations to federal lab patents.)

The incentives to patent and aggressively seek ways to commercialize EPB technology is influenced by factors similar to those operating in academic institutions. Like academics, NASA researchers’ careers are significantly shaped by publication record. So increased pressure to commercialize can conflict with publishing goals. One way this shows up is in attitudes among NASA researchers toward research focused on fundamental problems versus research directed at commercial, non-aerospace products. However, NASA researchers have some incentive to pursue commercialization because inventors receive one-fourth of any royalties from licensed patents. One major difference is that there appear to be fewer opportunities for consulting with private firms by NASA researchers – a major avenue available to university researchers for technology transfer.

According to interviews, patent citations to NASA-Lewis patents significantly underestimate the influence of EPB research for two possible reasons: (a) NASA may seek patents only when the technology relates to a government purpose (which would mean that contacting firms can acquire technology that would not involve a citation to EPB since no patent was applied for); and (b) NASA may not aggressively pursue litigation to prevent infringement on NASA patents if they view the technology as having been supported by the public and, therefore, not to be treated in the same as a private company’s technology. It is possible that companies may thus feel less compelled to cite NASA technology. Of course, citations would exist if the patent examiner at the USPTO is aware of NASA’s technology, whether in patent form or in another form of disclosure, such as in a publication.

Finally, EPB’s experience confirms the importance of geographic proximity, as well as the role of the larger "complex" of NASA states found in the geographic regression results. Using patent citations as the gauge, 20% of cites to EPB’s patents come from Ohio. This amount is higher than the same-state average for NASA (13%) as well as for the average for other federal labs (16% -- see Table 4). An additional 18% of cites come from the other "NASA Complex" states. Indeed, patent citations appear to understate the role of geography in EPB’s commercial impact. EPB has analyzed its relationships with private firms via collaborative arrangements such as Space Act agreements between July 1993 and July 1996. About 28% of these (25/90) were from Ohio, a rate somewhat higher than the fraction of citations, and roughly two-thirds of these were from EPB’s immediate region in Northeast Ohio.

More generally, technology transfer and knowledge transfer occur through many different mechanisms. Most citations to EPB patents did not involve what we have identified as direct contact, although there is somewhat more evidence of indirect communication. This is consistent with EPB’s findings when it asked firms that had contacted it how they learned of EPB; the answers were: through word of mouth (64%), by reading EPB publications (22%), and EPB presentations (6%).

There is some evidence that technology intermediaries may increase the efficiency of commercialization involving federal labs. Particularly where the technology is dual-use, a separate organization specializing in matching federal lab technology and industry need may be necessary to maximize economic benefits. For example, NASA has created six federally-funded Regional Technology Transfer Centers (RTTCs) to facilitate contact between business and the federal laboratory system. The RTTCs provide contact with all government labs and state & local science and technology organizations. The RTTC for NASA Lewis is the Great Lakes Industrial Technology Center (GLITeC) As an indication of its potential for influencing commercial innovation from federal lab technology, EPB has maintained records of contacts with approximately 400 companies over the past three years, the bulk of which initiated contact with EPB. In comparison with a previously-analyzed five-year period, during the last three years EPB submitted 90 proposed collaborative arrangements which have resulted in 43 awarded activities. The ratio of sharing of need and capabilities to initiating a proposed effort has increased by a factor of three from 30:1 to 9.3:1. The primary reason for the change has been GLITeC, which screens prospects and only encourages those companies representing a high probability of a good match between need and capability.

7. Conclusions and Implications