Comparing nationwide violent crime estimates: Summary-based vs. incident-based reporting

The FBI released official 2021 crime statistics last month, but the data leaves major gaps in our understanding of national-level crime rates. Unfortunately, the numbers are incomplete this year because 36% of U.S. law enforcement agencies, some of them major cities, failed to submit their crime reports. This is largely because the FBI made major changes to the way that it collects crime data, making the submission process much more complicated. Rather than following the arduous process, many agencies simply declined to participate in last year’s data collection. In comparison, 2020 crime statistics included data from 85% of U.S. law enforcement agencies.

This post briefly reviews agency participation rates and discusses what that means for our current understanding of national-level crime. Then, using data from the years 2017-2020 (all of the years for which data from both systems are available),  I examine whether the two systems produce similar violent crime estimates.

SRS versus NIBRS

Fewer agencies submitted their crime data in 2021 after the FBI finalized the transition from the historical Summary Reporting System (SRS) to the National Incident-Based Reporting System (NIBRS). Both systems are part of the Uniform Crime Reporting (UCR) program, each with their own strengths, though NIBRS is more complicated and has had consistently lower participation rates. But in January 2021, the FBI officially stopped accepting SRS submissions to facilitate a full transition to a NIBRS-only program. Thus, agencies had to choose to either submit their data through NIBRS or not at all. Instead of going through all the steps needed to submit to NIBRS, many agencies simply declined to participate.

At its inception in 1929, the UCR program consisted solely of the SRS and was very straightforward. As a summary-based system, it simply tabulated monthly counts of reported Part I crimes. This was ideal for providing a yearly snapshot of overall crime in the United States, but it was not detailed enough for in-depth understanding of spatial or temporal characteristics, offender and victim demographics, or weapons used.

In an effort to modernize the UCR, NIBRS was introduced in 1982. Compared to the SRS, NIBRS tracks information in much greater detail and covers more types of crimes, generating richer data and more opportunities for in-depth analysis. In theory, this should be preferable to the SRS, but it is unfortunately a lot less practical. While being more comprehensive, NIBRS is more complicated and labor-intensive, placing a greater burden on law enforcement agencies. In addition, many law enforcement agencies also needed to first make necessary upgrades to their data collection systems (e.g., upgraded technology, specialized staff) before adopting NIBRS.

Transition to a NIBRS-only System

NIBRS participation has always lagged. Even 30 years after its inception (2012), only 34.7% of agencies were reporting to NIBRS. In contrast, the SRS has regularly garnered participation rates of approximately 90% each year. Due to the higher participation rates, the SRS has always been more representative of our nation as a whole. Thus, it has historically been the preferred source of data for generating national-level crime estimates. For more information about the pros and cons of both systems, see my previous post.

To boost NIBRS participation and facilitate the eventual transition to NIBRS-only data collection, the FBI and BJS implemented the National Crime Statistics Exchange (NCS-X) Initiative. This project provided more than $166 million for research and development, implementation support teams, and direct support to individual agencies to help in the transition to NIBRS. The bulk of funds (more than $120 million) went directly to law enforcement agencies in the form of grants to cover expenses associated with NIBRS adoption (e.g. technological upgrades, specialized staff, necessary training).

A major part of the initiative was to increase the number of states that were “NIBRS-certified.” To achieve this goal, the FBI ultimately provided funding to the UCR programs for more than half of U.S. states. By April 2022, all state UCR programs were certified by the FBI to collect and submit NIBRS data.

Another major goal of NCS-X was to increase the number of local law enforcement agencies participating in NIBRS. Before deciding which agencies would receive funding, a sample of 400 scientifically selected eligible law enforcement agencies was drawn. These agencies, when combined with the then-current sample of NIBRS-contributors, would have generated a nationally representative sample. 72 of the agencies were large with more than 750 sworn officers, while 328 were small- to medium-sized with 750 or fewer officers. Eligible agencies were then able to apply for funding by submitting a project proposal detailing their transition plan and how they would use the money in the transition to NIBRS.

SRS and NIBRS Agency Participation Rates by Year

Based on participation data from the FBI’s Crime Data Explorer, the NCS-X project was moderately successful. From 2015 (the year funds were first awarded) through 2021, the number of agencies participation in NIBRS increased by a total of 25.5 percentage points, rising from 37.2% of agencies to 62.7%. Unfortunately, NIBRS participation is still somewhat low in comparison to the SRS, which regularly received at least 82% participation. While the SRS provided less detail, it was more sufficient for producing  better national-level crime estimates because it covered more agencies.

Even after NCS-X was implemented, NIBRS paticipation was very slow to take off. This is probably because there was a time lag between agency selection, disbursement of funds, and the eventual end goal of NIBRS adoption. Participation rates inched up increasingly year by year, finally surpassing the halfway mark by 2020 with 53.7% of agencies submitting data. In 2021, there was an even stronger push for NIBRS from the federal government due to their transition to a NIBRS-only system. Agency participation rates increased by nearly 10 percentage points that year, with approximately 62.7% of agencies submitting their data.

The figure below shows the total percentage of U.S. agencies (out of ~18,500 total) that participated in the SRS and NIBRS for the years 2011-2021.

Note: NIBRS and SRS aggregated participation rates were downloaded from the FBI’s Crime Data Explorer by state and year. For the years 2017-2021, detailed incident-level NIBRS data are available, which was used to cross-check the participation data for these particular years only. The detailed data contained some duplicates, and once removed, participation rates were about 2% lower than the rates shown in the aggregated numbers. For 2017-2021, the adjusted estimates are used.

Federal Funding Didn’t Help Everyone

Interestingly, some places that received federal funding via the NCS-X initiative were still not reporting as of 2021. Unfortunately some of these were major states and agencies, which creates substantial gaps in the data.

California received a cumulative of $31.5 million from the NCS-X project, $28.3m of which went toward individual agencies. But this didn’t seem to improve NIBRS participation very much, as only .018% of California agencies were participating in NIBRS by 2021. In comparison, in 2020 (before the SRS was retired), 82.1% of California agencies reported their crime data. Relatedly,  several of California’s major agencies were absent from the data completely despite receiving NCS-X grants, including Los Angeles City ($1.1m), Los Angeles County ($240k), and San Jose ($168k), which all serve populations in excess of 1m. Per BJS, San Jose’s transition is “in progress,” while Los Angeles County and City are both in the process of procuring new Records Mangement Systems. Some medium-sized agencies receiving funds also failed to report, including San Francisco ($5.3m), Riverside County ($1.5m), and Alameda County ($1.2m), among others. The money ($2.7m) seemed to help San Diego though, which was California’s only large agency (population 1.4 m) to report.

In Florida, despite the state receiving $17.6m in NCS-X funds, only two small tribal agencies submitted their crime data in 2021 (.002% of agencies). This is obviously in sharp contrast with Florida’s 2020 SRS participation rate of 81%. Needless to say, some major agencies (each serving populations of 900k or more) were missing, including Miami-Dade, Orange, and Hillsborough counties. Per correspondence with BJS, these agencies are all supposedly “working with the state UCR program to transition.”

New York received $11.2m in NCS-X funds, $10.3m of which went to individual agencies. Despite this, only 13.8% of agencies participated in NIBRS by 2021. In 2020 (prior to the transition), 56.9% of agencies reported to the SRS. Despite numerous attempts from the BJS, the largest department in the country, NYPD (serving 8.3 m), also has not yet reported to NIBRS. The two large neighboring agencies of Suffolk County (which received $2.4m, population 1.3m) and Nassau County (population 1m) also have not yet participated in NIBRS. Per correspondence with the BJS though, apparently Suffolk County’s transition is “almost complete,” and New York City is “transitioning on their own.” Less optimistically, Nassau County has “no transition information available.”

Pennsylvania received about $5.2m in NCS-X funds, $4.1m of which went to individual agencies. Despite this, only .022% of Pennsylvania agencies participated in the 2021 NIBRS data collection. In contrast, 78.8% of Pennsylvania agencies had reported to the SRS in 2020. Similar to the other states, the 2021 sample for Pennsylvania is missing major cities, such as Philadelphia (which received $3.9m in NCS-X funding, population 1.6m).

2021 NIBRS Population Coverage

Approximately 11,500 agencies reported their crime data to NIBRS in 2021. Together, the population served by these agencies represented 66% of the U.S. resident population. In comparison, the SRS used to garner responses from 16,000 agencies on average, typically achieving more than 90% population coverage. Of the approximately 7,000 law enforcement agencies that failed to submit their 2021 crime reports, a handful of these were pretty large agencies serving populations of 900,000 or more. Among the large agencies that are absent from the 2021 data are New York City (as well as Suffolk and Nassau counties), Los Angeles City (and Los Angeles County), San Jose, Phoenix, Tucson, and three major counties in Florida (Miami-Dade, Orange, and Hillsborough counties). Not suprisingly then, population coverage varied notably by region.

While the data in some states is fairly complete, the participation rates vary quite a bit. 32 states had excellent participation rates, where NIBRS-contributing agencies represented more than 90% of the state popluation (e.g., Texas, Washington, Montana, Ohio, Maine, and others). The estimates for these states are more likely to be accurate than the other states. 10 states had good participation rates, where NIBRS-contributing agencies represented between 60% and 85% of the state population (e.g., Louisiana, Indiana, Wyoming, Georgia, Nebraska, and others). The estimates for these states might be accurate, but should be considered preliminary and be interpreted with caution. 5 states had fair participation rates, where NIBRS-contributing agencies represented between 40% and 60% of the state population (i.e., Arizona, Alaska, New Jersey, Maryland, and Mississippi). The estimates for these states are less likely to be accurate and should also be interpreted with caution.

4 major states had very poor participation rates, where NIBRS-contributing agencies represented less than 20% of the population (i.e., California, Florida, New York, and Pennsylvania). The crime estimates for these states are unreliable and essentially useless. Gravely, these states are four of the five most populus states in the country, and their absence substantially affects the representativeness of this sample. In terms of population, California ranks first (39.5 million), followed by Texas (29.1 million), Florida (21.5 million), New York (20.2 million), and Pennsylvania (13.0 million). Of these five, Texas was the only one with decent participation rates, where reporting agencies represented 98% of the population. For California, Florida, New York, and Pennsylvania, responding agencies represented 7%, ~0.0%, 19%, and 17% of the state populations, respectively. Unfortunately, because the data is not representative of the country as a whole, we cannot draw any firm conclusions about national-level crime. 

See the figure below for a visual comparison of state-level population coverage.

NIBRS population coverage, 2021. Retrieved from the Bureau of Justice Statistics. Note: this data is based on the resident populations served by reporting agencies. Percentages reflect the percent of each state’s population that is represented by responding agencies.

Violent Crime Estimates: Comparing NIBRS and SRS data

Policymakers and researchers rely on these data to understand state and national crime trends, but that may be more challenging this year. Given the reduced participation rates discussed above, there are huge gaps in the data that affect the accuracy of nationwide crime estimates. Even more concerning, some of the states with moderately or highly questionable estimates have historically had violent crime rates that are much higher than the national average.

The FBI’s index of violent crime includes murder (and voluntary manslaughter), rape, robbery, and aggravated assault. In the recent years of 2015-2020, the average national violent crime rate (per the SRS) was 387.6 per 100,000 population. In contrast, California’s violent crime rates have been consistently higher than the national rate, with an average of 451.4 violent crimes per 100,000 each year. Similarly, Maryland’s violent crime rates from 2015-2020 are higher than the national average, with 473 violent crimes per 100,000. In New Mexico, the difference is even more astonishing, where the average violent crime rate from 2015-2020 was 773.8 per 100,000. Assuming that these patterns continued into 2021, it’s probable that NIBRS-generated crime rates underestimate the true volume of crime in the United States. This would be consistent with what we have seen in past years, comparing NIBRS and SRS.

Some of the major agencies (each serving populations exceeding 900k) that are omitted from the 2021 data also had violent crime rates that were higher than the national average. In 2021, the national average (computed by NIBRS) was 361.2 violent crimes per 100,000. In comparison, the NYPD’s 2021 CompStat data tabulated over 38,600 violent crimes, which equates to a rate of 465.6 per 100,000 based on their population served of 8.3 million people. Similarly, LAPD’s CompStat data tabulated over 30,000 violent crimes, which equates to a rate of 752.4 based on their population of 4 million. In San Jose, preliminary 2021 data suggest an average violent crime rate of 469.3 per 100,000. In Tucson, the preliminary police department data converts to a rate of 768 violent crimes per 100,000 (population served of 1.7 million). Needless to say, if these larger cities were included in the NIBRS data collection, it’s quite likely that the nationwide estimate of violent crime would exceed the NIBRS-computed rate of 361.2 per 100,000.

The figure below shows the SRS- and NIBRS-estimated number of violent crimes per 100,000 population for the years 2017-2020 (all years for which data for both systems are available). Estimates are regularly lower when calculated using NIBRS versus the SRS, suggesting that NIBRS data underestimates the volume of violent crime in the United States.