Lack of data hinders research on police violence: Official vs. open-source data

Over the last  few years, strain between police-community relations has intensified greatly as incidents have come to light (e.g., officer-involved shootings, protests) showing hostility between the police and the communities. Although circulation of “viral” videos of police incidents has increased, officer-involved shootings are still a relatively low-occurring event overall. Though, media exposure may make this problem seem more exacerbated than it is. Regardless, lack of transparency on official numbers contributes to contention over police. At present, there is no official number of the number of people killed by police annually.  Still, unreliable government numbers fail to portray the true scope of officer-involved shootings and fatalities. The National Use-of-Force Data Collection is the first national-level dataset to offer big-picture insights on police use of force but it is still in its infancy.

Otherwise, one of the closest things to an official source regarding officer-involved fatalities is collected by the U.S. National Vital Statistics System (NVSS), which tracks death certificates and causes of death in the country. Unfortunately though, these results may be undercounted by more than half, per a recent study in The Lancet. The study compared data from NVSS with data from three of four major publicly available, open-source databases compiled by various researchers (i.e., “Fatal Encounters,” “Mapping Police Violence,” and “The Counted”). The information in these databases is collated from news reports, department websites, and in some cases, public records requests. However, the data sources vary in where and how they collect their data (and how successful they were), so it is difficult to know whether any of these databases adequately represents the United States.

Update: As of December 9, 2021, law enforcement participation in the use-of-force data collection has remained quite low, which may cause the database to shut down.

The FBI created the National Use of Force Data Collection in 2015 in partnership with law enforcement agencies to provide nationwide statistics on law enforcement use-of-force incidents. As part of the project, a team of experts worked to develop universal agreement regarding what type of data should be collected and how it should be measured, with data collection beginning on January 1, 2019. The data collected includes information on law enforcement uses of force and basic information on the circumstances, subjects, and officers involved in these incidents, but it does not assess whether officers followed their department’s policy or acted lawfully. While participation in the data collection is voluntary, the FBI works closely with law enforcement agencies to encourage them to report their data. Initial participation data were reported when 40% of the total law enforcement officer population was reached (additional data will be released at 60% and 80% participation levels). As of September 2021, 7,559 out of 18,514 federal, state, local, and tribal law enforcement agencies throughout the nation had participated and provided use-of-force data. The officers employed by these agencies represented 54% federal, state, local, and tribal sworn officers in the nation. However, actual data on shootings/fatal incidents by police have not been released, likely because the sample is not representative enough to present valid estimates at this time. The parameters needed to create valid estimates depend on the coverage rate for each state.

As stated above, the U.S. National Vital Statistics System tracks death certificates and causes of death in the country, and data from coroners and medical examiners is collected regarding type of death, which sometimes will specify whether a police shooting was related to the death. The most recent estimates are from 2019, but unfortunately, the results regarding police shootings may be undercounted by more than half. In police-related deaths, a medical examiner or coroner must fill out the cause of death on the death certificate, however, their training and expert opinions can still vary, and they may not mention whether police violence contributed to the death. Also, there are sometimes conflicts of interest between police departments and medical examiners, which might also contribute to the undercount. In 2019, the database had tabulated 142,440 deaths (a rate of 2.6 per 100,000), and the data showed  14,730 firearm deaths (a rate .05 per 100,000). The data showed only 5 deaths by “official authorities” (rate is not reported as it is unreliable). Additionally, there were 662 handgun-related deaths with “unspecified intent,” so it is unclear whether this could have been related to authorities as well.

Due to the issues in the current official data, independent researchers have taken it upon themselves to collate information on police-related fatalities using information from news reports, department websites, and public records requests. Four major non-governmental, open-source databases on police killings include “Fatal Force,” “The Counted,” “Mapping Police Violence,” and “Fatal Encounters.” However, the data sources vary in where and how they collect their data, which makes it difficult to know which one is the most accurate. For one, most of these will suffer from selection bias as they will only include police shootings/fatalities that are reported in the media. For two, many of the data sources vary in terms of how successfully they were able to collect data, which also may affect results.

Fatal Force was collated by the Washington Post on data regarding fatal police shootings from 2015 through 2021, including data from a variety of important case characteristics (e.g., race, gender) that can be broken down by city and state. The project has tallied more than 6,700 events from 2015-2021 and 915 in 2021 so far (a rate of 0.24 per 100,000).  The Post documents only shootings where an on-duty police officer shoots and kills a civilian, and does not track deaths of people in police custody or non-shooting deaths. The Post’s data relies primarily on news accounts, social media postings, and police reports. From 2015-2020, it appears that fatal police shootings remained relatively stable with an average of 991 per year (a rate of .30 per 100,000), ranging from a low of 957 in 2016 and a high of 1,021 in 2020. People were armed in an average of 93.5% of cases from 2015-2021, 60.5% of whom were armed with a gun. Per the NVSS, this represented less than .04% of total deaths on average.

The Counted was collated by researchers at the Massachusetts Institute of Technology (MIT). This project tracked fatal police violence in America in 2015 and 2016 by pulling data from police reports, local governments, news and research organizations, and open-source reporting projects. The Counted also encouraged crowdsourced accounts from witnesses, who can contact reporters and send files anonymously. The project identified 1,146 police killings in 2015 (a rate of .36 per 100,000) and 1,093 in 2016 (a rate of .34 per 100,000), but they do not present data for the years 2017 forward. While data on whether subjects were armed was limited to 2016, it showed that 84.5% of cases involved armed subjects. This dataset tried to count every death (though most of the deaths were officer-involved shootings) The data offers snapshot descriptions of the killings, detailing a wide range of circumstances that can be broken down state-by-state.

Mapping Police Violence has gathered information from 13,147 police departments and 2,878 sheriff’s departments across the country from 2013-2020, and data can be broken down by state and police department.  In some cases, researchers were able to obtain data on individual-level police officers, including things such as uses of force and disciplinary records. From 2013-2020, the researchers tabulated information on 9,693 police-related fatalities in the U.S. As of November 2021, the data has tabulated 925 police killings so far this year (a rate of .28 per 100,000) . These numbers are approaching those similar to past years, with an average of 1,096 from 2013-2020 (a rate of .34 per 100,000), ranging from a low of 1,049 in 2014 to a high of 1,144 in 2018. The 2020 report found that 1,127 people were killed by the police in 2020, the majority (96%) of which were from gunshot injuries. 120 people (10.6%) were killed after police stopped them for a traffic violation, and 97 people (8.6%) were killed after police responded to reports of someone behaving erratically or having a mental health crisis.  People were armed in approximately 90% of cases, 53.6% of whom were armed with a gun. This data also suggested that rates of violent crime in cities did not determine rates of killings by police. For example, Buffalo and Newark police departments had relatively low rates of police violence despite high crime rates while Spokane and Orlando had relatively low crime rates and high rates of police violence. On the website, you can view nationwide trends, state-by-state trends, city-level trends, and the 2020 final report.

Fatal Encounters, another open-source database, tracks U.S. police killings from 2000-2021 by city, state, and county. This data is likely more comprehensive than some of the others because it attempts to actually collect data from police departments (via public records requests, etc.). However, one limitation is that police departments don’t always respond, reports contain inconsistent amounts of information, and many departments redact information on important variables, leading to a fair amount of missing data. Nonetheless, it does appear to have higher numbers of deaths compared to other the other publicly-available datasets. For example, in 2020, Fatal Encounters tabulated 2,085 deaths while Mapping Police Violence tabulated 1,127. In 2016, Fatal Encounters tabulated 1,609 deaths, compared to 1,093 in The Counted and 1,070 in Mapping Police Violence. As of October 2021, the project has tallied 31,075 deaths from 2000 to 2021 and a total of 1,579 in 2021 so far (a rate of .47 per 100,000). From 2000-2020, there was an average of 1,405 police killings per year (a rate of .45 per 100,000), ranging from a low of 865 in 2000 to a high of 2,085 in 2020. While data on weapon type and whether subjects were armed was limited to approximately 13,800 cases out of 31,075, it nonetheless showed that people were armed in 60.5% of cases and 45.3% of these cases involved a firearm. While this data source definitely has limitations, it appears to be the most comprehensive at this time (and it also covers the longest time span). It does suggest that overall the rates of fatal encounters has increased over time though, from a low of 865 (a rate of .30 per 100,000) in 2020 to a high of 2,085 (a rate of .63 per 100,000) in 2021. However, it also could also be a result of improvements in documentation and may not represent a “true” increase.

The lack of data available regarding police uses of force inhibits the ability to study police misconduct more generally, as well as the degree to which officers are held accountable. At present, there are some tools that police departments used that can be used to identify police misconduct and ensure accountability for problematic behavior instead of exclusively focusing on shooting incidents. For example, many departments have implemented “Early Warning Systems” (EWS), which is a data-based police management tool designed to identify officers whose behavior is problematic. As an early response, the system alerts the department to these individuals and warns the officers while providing counseling or training to help change their problematic behavior. Most agencies will also monitor that officer’s performance after the initial intervention. Unfortunately, EWSs still need refinement to improve their accuracy, as research is still mixed. For example, a 2020 research review found mixed results on the effectiveness of EWSs, depending on the outcome measure used. EWSs seemed effective in reducing complaints and uses of force, but other studies found mixed results or no effects on officer behavior or performance, and others even found a decline in officer productivity. Relatedly, a 2021 study found that EWSs may not be that accurate in actually differentiating “problem” from “non-problem” officers. They looked at police reports from citizen interactions and compared performance between EWS-flagged officers and non-flagged officers. There was not a statistically significant difference between EWS-flagged and non-flagged performance, suggesting that the “thresholds” being used to identify problem officers may not be working effectively. Unfortunately, this type of data is typically restricted to police departments and is not typically accessible to the public.

However, one database that might be able to be leveraged to look at police accountability is the National De-certification Index (NDI), which is ran by the International Association of Directors of Law Enforcement Standards and Training (IADLEST), an organization dedicated to the improvement of public safety personnel. The database was created with intention for use by law enforcement agencies for the purpose of hiring authorities to conduct background investigations on potential candidates. The information is collected from certifying agencies in each state (in most cases, the State Peace Officer Standards and Training agency [POST]). It does not examine police use of force or shootings specifically, but it collects information on officers’ disciplinary infractions and their misconduct histories. From the data reported by 46 certifying agencies, the findings that sufficiently meet a state’s official sanction of misconduct are accepted into the NDI. Regardless of the original intention for the data, it is possibly a source that can be leveraged for research, particularly if combined with other data on police use of force. The data is not publicly accessible, but application requests can be made for access.

In sum, higher-quality data is needed to guide this important issue that continues to divide the country. Unfortunately at this time, it is hard to get an accurate big-picture look regarding police use of force in the United States. Thankfully, the approaches described above do represent progress in terms of data collection on police use of force, and hopefully will improve as more agencies participate in the FBI Use-of-Force task force reporting system. Unfortunately, law enforcement participation in the use-of-force data collection has been limited, which may cause the database to shut down.