7  Linking databases

Almost any empirical research project in accounting or finance research will involve merging data from multiple sources. When we joined data tables in Chapter 2, we had a common identifier across tables. But this will not always be the case. For example, to evaluate the market reaction to earnings announcements, we might start with data on comp.fundq from Compustat, where gvkey and datadate can be used to identify firm-quarters and associated announcement dates (rdq), and then look to merge with daily stock return data from the Center for Research in Security Prices (CRSP, pronounced “crisp”) on crsp.dsf, which uses permno and date to identify each firm’s trading equity and daily stock returns (ret). But this raises the question: Which permno (if any) matches a given gvkey?

Alternatively, given security price information from CRSP, we might want to know what is the most recent financial statement information for that security. In other words, which gvkey (if any) matches a given permno? This chapter provides guidance on the standard approaches to answering these questions with a focus on linking CRSP and Compustat.

Tip

The code in this chapter uses the packages listed below. For instructions on how to set up your computer to use the code found in this book, see Section 1.2. Quarto templates for the exercises below are available on GitHub.

7.1 Firm identifiers

The idea behind a firm identifier is that it uniquely identifies a firm for a particular purpose. While Compustat uses GVKEYs, CRSP uses PERMNOs, the SEC uses CIKs, stock exchanges use tickers, and there are also CUSIPs.1 Table 7.1 lists some common firm (and security) identifiers. Of course, identifiers apply not only to firms, but also people. Nonetheless, most of this chapter focuses on firm identifiers, in part because of the importance of firms as units of observation in accounting and finance research, but also because identifying firms is much harder than identifying people.2 While not specific to the platform we describe in this book, the issue of identifiers is one that seems less perplexing and better-handled when a relational database provides the backbone of your data store as it does here.

Table 7.1: Firm and security identifiers
Data provider Firm identifiers Notes
Compustat (comp) gvkey
CRSP (crsp) permno, permco permno is a security identifier
IBES (ibes) ticker ticker is not necessarily the ticker assigned to the firm by the exchange on which it trades
SEC EDGAR CIK
Audit Analytics (audit) company_fkey company_fkey is the same as CIK
Various CUSIP CUSIP is a security identifier

7.1.1 Firm identifiers: A quiz

Our sense is that firm identifiers is one of those topics that seem easy, but is actually fairly tricky. Here is a quick quiz to test your knowledge of firm identifiers.3

  • General Motors Corporation declared bankruptcy in June 2009. Does the “successor firm” General Motors Company have the same GVKEY as General Motors Corporation? The same PERMNO?
  • Can a CUSIP map to more than one PERMNO? To more than one GVKEY?
  • Can a PERMNO map to more than one CUSIP?
  • Can a GVKEY map to more than one PERMCO?
  • Can a PERMCO map to different CIKs?
  • If you have two data sets, \(X\) and \(Y\) and CUSIP is a “firm” identifier on each, can you simply merge using CUSIPs?
  • When would a “firm” change CUSIPs?
  • When would a “firm” change CIKs?
  • If the firm identifier on IBES is ticker, should I merge with CRSP using ticker from crsp.stocknames?

Maybe you know the answers to some questions, but not all. If so, read on; this chapter aims to provide answers to many of these questions.4

7.2 The CRSP database

According to its website, “the Center for Research in Security Prices, LLC (CRSP) maintains the most comprehensive collection of security price, return, and volume data for the NYSE, AMEX and NASDAQ stock markets. Additional CRSP files provide stock indices, beta-based and cap-based portfolios, treasury bond and risk-free rates, mutual funds, and real estate data. [CRSP] maintains the most comprehensive collection of security price, return, and volume data for the NYSE, AMEX and NASDAQ stock markets. Additional CRSP files provide stock indices, beta-based and cap-based portfolios, treasury bond and risk-free rates, mutual funds, and real estate data.” We will discuss the CRSP/COMPUSTAT Merged Database in Section 7.3. CRSP documentation provides details on other CRSP databases, such as CRSP US Treasury and Inflation Series, CRSP Mutual Funds, and CRSP/Ziman Real Estate Data Series.

CRSP provides PERMNO, its own “permanent identifier” for each security in its database. Additionally, CRSP provides a company-level identifier, PERMCO, for each company. WRDS tells us that CRSP’s goals in creating these identifiers are to allow “for clean and accurate backtesting, time-series and event studies, measurement of performance, accurate benchmarking, and securities analysis.”

According to WRDS, “CRSP contains end-of-day and month-end prices on all listed NYSE, Amex, and NASDAQ common stocks along with basic market indices, and includes the most comprehensive distribution information available, with the most accurate total return calculations.” We create remote data frames for crsp.dsf (end-of-day prices) and crsp.msf (month-end prices).

db <- dbConnect(RPostgres::Postgres(), bigint = "integer")

dsf <- tbl(db, Id(schema = "crsp", table = "dsf")) 
msf <- tbl(db, Id(schema = "crsp", table = "msf"))
db <- dbConnect(duckdb::duckdb())

dsf <- load_parquet(db, schema = "crsp", table = "dsf")
msf <- load_parquet(db, schema = "crsp", table = "msf")

Let’s look at a few rows from crsp.dsf.

dsf |> collect(n = 5)
# A tibble: 5 × 20
  cusip    permno permco issuno hexcd hsiccd date       bidlo askhi   prc
  <chr>     <int>  <int>  <int> <int>  <int> <date>     <dbl> <dbl> <dbl>
1 68391610  10000   7952  10396     3   3990 1986-01-07  2.38  2.75 -2.56
2 68391610  10000   7952  10396     3   3990 1986-01-08  2.38  2.62 -2.5 
3 68391610  10000   7952  10396     3   3990 1986-01-09  2.38  2.62 -2.5 
4 68391610  10000   7952  10396     3   3990 1986-01-10  2.38  2.62 -2.5 
5 68391610  10000   7952  10396     3   3990 1986-01-13  2.5   2.75 -2.62
# ℹ 10 more variables: vol <int>, ret <dbl>, bid <dbl>, ask <dbl>,
#   shrout <dbl>, cfacpr <dbl>, cfacshr <dbl>, openprc <dbl>,
#   numtrd <int>, retx <dbl>

The CRSP Indices database contains a number of CRSP indices. Here we focus on two index tables, crsp.dsi and crsp.msi, which can be viewed as complementing crsp.dsf and crsp.msf respectively.

dsi <- tbl(db, Id(schema = "crsp", table = "dsi")) 
msi <- tbl(db, Id(schema = "crsp", table = "msi"))
dsi <- load_parquet(db, schema = "crsp", table = "dsi")
msi <- load_parquet(db, schema = "crsp", table = "msi")

7.2.1 Exercises

  1. Looking at crsp.dsf and crsp.msf, we see that prc can be negative. Do negative stock prices make sense economically? What do negative stock prices on CRSP mean?5 What would be an alternative approach to encode this information? (Write code to recast the data using this approach.) Why do you think that CRSP chose the approach it uses?

  2. How do ret and retx differ? Which variable are you more likely to use in research?

  3. Is the date variable on crsp.msf always the last day of the month? If not, why not?

  4. Suggest the “natural” primary key for crsp.dsf and crsp.msf. Check that this is a valid primary key for crsp.msf.

  5. In the code below, we are using collect() followed by mutate(month = floor_date(date, "month")) to calculate month. What changes occur in terms of where the processing happens if we replace these two lines with mutate(month = as.Date(floor_date("month", date))) |> collect()? Do we get different results? What effect does as.Date() have?

plot_data <-
  dsf |>
  select(date) |>
  filter(between(date, "2017-12-31", "2022-12-31")) |>
  collect() |>
  mutate(month = floor_date(date, "month"))
  1. What is being depicted in Figures 7.1 and 7.2? What are the sources of variation across months in the Figure 7.1? Can you guess what is the main driver of variation in Figure 7.2? Create an additional plot to visualize the source of variation in Figure 7.1 not depicted in Figure 7.2.
plot_data |>
  count(month) |>
  ggplot(aes(x = month, y = n)) +
  geom_bar(stat = "identity") +
  scale_x_date(date_breaks = "2 months", expand = expansion()) +
  theme(axis.text.x = element_text(angle = 90))
A plot of the number of observations on the CRSP daily stock file by month for years 2018 through 2022. Values fluctuate from 150,000 to over 200,000 and tend to increase over time. Plot is part of an exercise assigned to students to relate what is shown to the underlying data.
Figure 7.1: Number of observations by month (#1)
plot_data |>
  distinct() |>
  count(month) |>
  ggplot(aes(x = month, y = n)) +
  geom_bar(stat = "identity") +
  scale_x_date(date_breaks = "2 months", expand = expansion()) +
  theme(axis.text.x = element_text(angle = 90))
An alternative plot of the number of observations on the CRSP daily stock file by month for years 2018 through 2022. Values fluctuate from 19 to 23 and have no apparent trend over time. Plot is part of an exercise assigned to students to relate what is shown to the underlying data.
Figure 7.2: Number of observations by month (#2)
  1. What is the primary key for crsp.dsi and crsp.msi? Verify that it is a valid key for both tables.

  2. Using the dplyr verb anti_join(), determine if there are any dates on crsp.dsf that do not appear on crsp.dsi or vice versa. Do the same for crsp.msi and crsp.msf.

7.3 Linking CRSP and Compustat

The CRSP/Compustat Merged (CCM) database provides the standard link between CRSP data and Compustat’s fundamental data. The CCM provides three tables that you will see in common use:

  • crsp.ccmxpf_lnkhist
  • crsp.ccmxpf_lnkused
  • crsp.ccmxpf_linktable

The reality is that the only table we need to worry about is crsp.ccmxpf_lnkhist, as the other two tables can be (and likely are) constructed from it (see here for details).6

ccmxpf_lnkhist <- tbl(db, Id(schema = "crsp", table = "ccmxpf_lnkhist"))
ccmxpf_lnkhist <- load_parquet(db, schema = "crsp", table = "ccmxpf_lnkhist")

7.4 All about CUSIPs

According to CUSIP Global Services, “CUSIP identifiers are the universally accepted standard for classifying financial instruments across institutions and exchanges worldwide. Derived from the Committee on Uniform Security Identification Procedures, CUSIPs are 9-character identifiers that capture an issue’s important differentiating characteristics for issuers and their financial instruments in the U.S. and Canada.”

CUSIP Global Services uses the CUSIP of Amazon.com’s common stock, 023135106, as an example of the components of a 9-character CUSIP. The first six characters—023135—represent the issuer, which is Amazon.com. While in this case the issuer is a company, an issuer could be a municipality or a government agency. The next two characters (10) indicate the type of instrument (e.g., debt or equity), but also uniquely identifies the issue among the issuer’s securities. The final character (6) is a check digit created by a mathematical formula. This last character will indicate any corruption of the preceding 8 characters. Note that the characters need not be digits. For example, according to an SEC filing, the Class C Common Stock of Dell Technologies Inc. has a CUSIP of 24703L202, which contains the letter L.

While a full CUSIP always comprises nine characters, many data services abbreviate the CUSIP by omitting the check digit (to create an “eight-digit” CUSIP) or both the check digit and the issue identifier (to create a “six-digit” CUSIP). For example, the CRSP table crsp.stocknames uses eight-digit CUSIPs.

Notwithstanding the existence of crsp.ccmxpf_lnkhist, some researchers choose to link CRSP and Compustat using CUSIPs. For example, the code supplied with Jame et al. (2016) merges CRSP daily data on returns, prices, volume and shares outstanding with Compustat data on shareholders’ equity using CUSIP and “year”.11

To evaluate the appropriateness of using CUSIPs to link CRSP and Compustat, we can construct a link table for comp.funda using CUSIPs (funda_cusip_link below) and compare with with an analogous link table constructed using ccm_link (funda_ccm_link below). First, let’s construct the subset of comp.funda of interest.

funda_mod <-
  funda |>
  filter(indfmt == "INDL", datafmt == "STD",
         consol == "C", popsrc == "D") |>
  mutate(mkt_cap = prcc_f * csho) |>
  select(gvkey, datadate, cusip, at, mkt_cap) 

Our source for PERMNO-CUSIP links is crsp.stocknames. There are some cases where there is no value on ncusip, but there is a value on cusip and we use coalesce() to fill in missing values in such cases.

stocknames <- tbl(db, Id(schema = "crsp", table = "stocknames"))
stocknames <- load_parquet(db, schema = "crsp", table = "stocknames")
stocknames_plus <-
  stocknames |>
  mutate(ncusip = coalesce(ncusip, cusip))

Now we can construct funda_cusip_link containing CUSIP-based matches for each (gkvey, datadate).

funda_cusip_link <-
  funda_mod |>
  mutate(ncusip = str_sub(cusip, 1L, 8L)) |>
  inner_join(stocknames_plus, 
             join_by(ncusip, 
                     between(datadate, namedt, nameenddt))) |>
  select(gvkey, datadate, permno, permco)

Similarly, we can construct our matches using ccm_link.

funda_ccm_link <-
  funda_mod |>
  select(gvkey, datadate) |>
  inner_join(ccm_link, 
             join_by(gvkey, 
                     between(datadate, linkdt, linkenddt))) |>
  select(gvkey, datadate, lpermno, lpermco) |>
  rename(permno = lpermno, permco = lpermco)

Finally, we combine both sets of matches in funda_link_combined for comparison.

funda_link_combined <-
  funda_mod |>
  select(-cusip) |>
  left_join(funda_ccm_link, by = join_by(gvkey, datadate)) |>
  left_join(funda_cusip_link,
            by = join_by(gvkey, datadate), 
            suffix = c("_ccm", "_cusip")) |>
  mutate(same_permno = permno_ccm == permno_cusip,
         same_permco = permco_ccm == permco_cusip,
         has_permno_ccm = !is.na(permno_ccm),
         has_permno_cusip = !is.na(permno_cusip)) |>
  filter(has_permno_ccm | has_permno_cusip) |>
  collect()

Regarding Table 7.9, we can probably view the cases with same_permno as valid matches, but would probably need to check the cases where same_permno is FALSE.

funda_link_combined |>
  count(same_permno, same_permco)
Table 7.9: Comparison of CCM- and CUSIP-based links
same_permno same_permco n
FALSE FALSE 131
FALSE TRUE 160
TRUE TRUE 242848
NA NA 118509

The cases where same_permno is NA in Table 7.9 are explored in Table 7.10. We would need to investigate the cases where one of permno_ccm or permno_cusip is NA to understand the source of the non-matches in one table or the other. However, a reasonable view seems to be that ccm_link provides many valid matches that are lost when matching using CUSIPs and for this reason crsp.ccmxpf_lnkhist should be preferred to CUSIP-based matches.

funda_link_combined |>
  count(has_permno_ccm, has_permno_cusip)
Table 7.10: Differences in coverage of CCM- and CUSIP-based link tables
has_permno_ccm has_permno_cusip n
FALSE TRUE 3233
TRUE FALSE 115276
TRUE TRUE 243139

7.4.1 Exercises

  1. Is there any evidence of “reuse” of CUSIPs on crsp.stocknames? In other words, are there any ncusip or cusip values associated with more than one permno?

  2. The CRSP table crsp.stocknames includes two CUSIP-related fields, cusip and ncusip. What are the differences between the two fields? What does it mean when ncusip is missing, but cusip is present?

  3. Like CUSIPs, PERMNOs are security-level identifiers. Can a PERMNO be associated with more than one CUSIP at a given point in time? Can a PERMNO be associated with more than one CUSIP over time?

  4. Looking at entries on crsp.stocknames where ticker is DELL, we see two different permno values. What explains this?

stocknames |> 
  filter(str_detect(comnam, '^DELL ')) |> 
  select(permno, cusip, ncusip, comnam, siccd, namedt, nameenddt)
permno cusip ncusip comnam siccd namedt nameenddt
11081 24702R10 24702510 DELL COMPUTER CORP 3570 1988-06-22 2003-07-21
11081 24702R10 24702R10 DELL INC 3570 2003-07-22 2013-10-29
16267 24703L10 24703L10 DELL TECHNOLOGIES INC 3824 2016-09-07 2018-12-27
18267 24703L20 24703L20 DELL TECHNOLOGIES INC 3824 2018-12-28 2020-03-22
18267 24703L20 24703L20 DELL TECHNOLOGIES INC 3571 2020-03-23 2023-12-29
  1. Looking at permno of 11081 (Dell), we see two different CUSIP values. What change appears to have caused the change in CUSIP for what CRSP regards as the same security?

  2. Choose a row from funda_link_combined where same_permco is FALSE. Can you discern from the underlying tables what issue is causing the difference and which match (if any) is valid? (Hint: Do rows where gvkey %in% c("065228", "136265") meet this condition? What appears to be the issue for these GVKEYs?) Can you conclude that the CCM-based match is the preferred one in each case?

  3. Choose a row from funda_link_combined where has_permno_cusip is TRUE and has_permno_ccm is FALSE. Can you discern from the underlying tables whether the CUSIP-based match is valid? (Hint: Do rows where gvkey %in% c("033728", "346027") meet this condition? What appears to be the issue for these GVKEYs?)

  4. Given the results shown in Table 7.9 and Table 7.10 and your answer to the previous two questions, can you conclude that the CCM-based match is preferred to the CUSIP-based match in each case?


  1. Of course, PERMNOs, CUSIPs, and tickers (at best) identify securities, not firms. More on this below.↩︎

  2. In the United States, a Social Security Number (SSN) is a pretty robust identifier of people, as would be a Tax File Number (TFN) in Australia. Though, as researchers, we generally don’t have access to SSNs or TFNs.↩︎

  3. Obviously, we are assuming that you recognize the various identifiers. If not, read on.↩︎

  4. Coverage of CIKs is deferred to Chapter 23 and we do not use IBES data in this book.↩︎

  5. CRSP documentation can be found at https://go.unimelb.edu.au/gcd8.↩︎

  6. WRDS says “SAS programmers should use the Link History dataset (ccmxpf_lnkhist) from CRSP”: https://go.unimelb.edu.au/akw8.↩︎

  7. The meaning of a linktype of NP is unclear, as no documentation of this code seems to exist.↩︎

  8. A careful reader might have noticed that we actually already use a window function in Chapter 2, namely fill(). However, there we used arrange() instead of window_order(). The window_order() function is only available for remote data frames because it provides functionality not available with local data frames. Readers coming from an SQL background might observe that dplyr’s group_by() is “overloaded” in the sense that it does the work of both the GROUP BY statement and the PARTITION BY clause in SQL. A short discussion of window functions is found in Chapter 21 of R for Data Science.↩︎

  9. See https://go.unimelb.edu.au/z9w8 for details.↩︎

  10. The same seems to be true for the case with gvkey of 003581. Again, it’s not clear why CRSP switched the primary permno on 2018-01-01.↩︎

  11. Only the last observation for a calendar year is kept by Jame et al. (2016) for CRSP and year means year(datadate) for Compustat.↩︎