Misleading “study” claims that DA Schubert’s term led to increased crime

A new report released last week by the Center on Juvenile and Criminal Justice (CJCJ) by Mike Males compares crime trends between Sacramento and San Francisco. The report’s narrative is fairly evident in the title: “Tough Talking Sacramento District Attorney Presides Over Homicide and Violence While Liberal San Francisco Enjoys Major Decreases.” To summarize, the report claims that Sacramento has seen major increases in homicide and violence while under the leadership of a conservative District Attorney Anne Marie Schubert, while San Francisco has seen major decreases in crime due to their progressive reforms and liberal District Attorney Chesa Boudin.

In other words, CJCJ concludes that liberal leadership reduces crime while conservative leadership increases it. But the report is riddled with methodological shortcomings that render it meaningless in being able to determine cause and effect. In this post, I review the key findings of the report and outline the methodological problems that undermine these findings.

Chesa Boudin became San Francisco’s district attorney in January 2020. Prior to 2020, the city of San Francisco was already under liberal leadership, so the new leadership in 2020 did not mark a shift in political affiliation. Anne Marie Schubert became Sacramento’s district attorney in 2014. Prior to 2014, the city of Sacramento was under liberal leadership, so this did mark a shift in political affiliation from liberal to conservative. The report claims that from 2014-2021, after Schubert’s term began, “rates of homicide, other violent crimes, and property crimes fell faster in San Francisco than Sacramento.”

One point that is very important to mention is that correlation does not equal causation. Correlation is relatively easy to prove, though proving causation is much harder. The three criteria needed to determine causality are: 1) an association between two factors, 2) proper temporal order, i.e., that the cause preceded the effect, and 3) elimination of alternative explanations for the outcome of interest. Technically the CJCJ report met the first criteria by identifying a weak association between two factors: that is, he showed that the conservative DA in Sacramento was correlated with rises in violent crime. The second two criteria are not met in the report, though. Further, there are some other questionable points about the analysis in general.

Problem #1: The regression analysis is not rigorous and lacks “goodness of fit” statistics.

Violent crime rates, homicide rates, and property crime rates are displayed in the CJCJ report using line graphs. The author took the rates for each year from 2014-2021 and then fitted a line to attempt to capture the relationship over the years. While this line is based on all of the data points, the author did not look at year-to-year trends. He explained that this approach is better than “cherry-picking” certain years because it uses a regression line that incorporates all of the data points. In some ways, this is a good approach. However, it’s not a good approach if the data do not follow a linear pattern. If data do not follow a linear pattern, then fitting a linear regression line will contain a large margin of error that might render it meaningless. Thus, researchers typically run a few different models and compare “goodness of fit statistics” before selecting a final model.

These statistics should be reported so that readers can understand whether the model was a good fit to the data. In this case, there is a strong possibility that the linear regression fit contains a wide margin of error (as a rough guide, the further away that data points are from the best fit line, the more error there is; see Figures 1-3 of the CJCJ report for more information). However, authors do not present information on the error or fit statistics of the regression line, so this cannot be assessed. The author also does not statistically control for the impact of important variables in the analysis, despite the fact that this is relatively easy to do with a regression analysis. This limited his ability to eliminate alternative explanations for the outcome, which I discuss in further detail below.

When it comes to temporal trends, a more methodologically sound approach is a time series analysis. When there is a specific time point of interest (in this case, 2014), researchers can conduct an “interrupted time series” that compares temporal trends from before and after a certain cut point. The time series approach is preferred for looking at changes over time, especially when the rates ebb and flow slightly each year.

Problem #2: There is no pre-intervention data, so the author cannot determine whether the cause actually preceded the effect.

Another major problem is that there is no pre-intervention data (sometimes referred to as “baseline data”). According to the report: “comparing Sacramento to San Francisco, rates of violence rose 9% in Sacramento from 2014-2021, while rates in San Francisco decreased by 29% from 2014-2021.” The author claims this is due to the fact that DA Anne Marie Schubert took office in 2014. To ascertain temporal order, the author must show that the cause (DA Schubert’s term) preceded the effect (changes in crime trends). However, there is no way to determine whether crime trends actually changed, and if so, whether change occurred specifically after 2014 or whether it started earlier.Without having any data on the period preceding Schubert’s term, it isn’t possible to know that Sacramento’s upward trend was a result of Schubert. It is possible that one or both cities’ crime trends were already following a certain path, and might be part of a longer-term trajectory that is not captured in the number of years presented.

In the context of this study, Sacramento’s crime should have begun to increase after 2014, and should not be part of a longer-term pre-existing trend. However, this can’t be determiend because the author does not examine data from pre-2014. Thus, the author is unable to prove that crime trends changed after 2014. He presented eight years of data, so ideally the pre-period data would also go back eight years (to 2006). To get a sense of the pre-2014 trends, I took a look at the UCR data for San Francisco and Sacramento for the years 2006-2019 (unfortunately, I had to exclude 2020-2021 because they are not available through the UCR, and the numbers presented on police department websites are tabulated differently than historical data).

Reviewing the trends from 2006-2013 (before Schubert’s term) revealed that violent crime rates were consistently higher in Sacramento most years, with the exception of 2013 (see figure below). Conversely, after 2013, San Francisco’s violent crime rate actually climbed higher than Sacramento’s, and remained higher for all but one year (with the exception of 2016). While both cities have seen decreases in violent crime over the last couple decades, San Francisco’s violent crime rate has been higher than Sacramento’s for almost every year since Schubert’s term began. If changes in violent crime were truly due to Schubert’s leadership, then one would expect to see the opposite. That is, one would expect San Francisco’s post-2014 rates to be lower than Sacramento’s.

Looking at Sacramento, violent crime rates were highest in 2006, and have consistently declined through 2019, with brief upticks in 2012 (about 40 additional crimes per 100,000) and 2015 (about 115 additional crimes per 100,000). If any changes in crime were really due to the change in leadership, one would expect to see a sharp change beginning in 2014 that persisted over time, rather than a brief spike. San Francisco has followed a similar trend, with violent crime rates being the highest in 2006, followed by consistent declines throughout 2019, with brief upticks in 2012 (about 5 additional crimes per 100,000), 2013 (about 140 additional crimes per 100,000), and 2017 (about 5 additional crimes per 100,000). Both cities seem to be on an overall downtrend that began potentially as early as 2006, though San Francisco had a brief spike in 2013 and Sacramento had a brief spike in 2015. Otherwise, the downward trend seemed to begin awhile ago, without major changes in its overall trajectory.

Problem #3: The author does not assess differences between the two cities that could be impacting the outcome.

Pre-intervention data is also useful to show that the cities were similar to each other before Schubert’s term (i.e., at “baseline”). The author did not assess pre-intervention charateristics of either city at the outset, which severely limited his ability to ascertain cause and effect. The two cities should look similar to each other at the outset (in other words, they should have “baseline equivalence“) in terms of demographics, existing crime rates, urbanicity, income, and other factors. If they don’t, then the author would need to statistically control for any differences between groups. For example, if one city has a higher poverty rate, then poverty rate can be statistically controlled for in the analysis, which is easy to do with a linear regression model. This is especially critical when comparing one city to one city, as there are no other cities in the group to balance out potential outliers. Without pre-intervention data though, the author could not show that groups were similar to each other. Likewise, he could not control any pre-existing differences between groups, and it is possible that other differences between cities (aside from the DAs) were driving outcomes. Thus, the author did not eliminate alternative explanations for the outcome.

One example of how the cities clearly differed from each other is that San Francisco has a “higher retail density and tourist visitation” than Sacramento. It’s unclear why the author did not statistically control for retail density, when he knows that it differs between the cities. This would have been easy to do with a regression analysis. Nonetheless, the author uses this to excuse San Francisco’s higher rates of robbery, burglary, and thefts. This is a great example of where the author should have controlled for the impact of retail density in the analysis, rather than just relying on speculation.

Problem #4: The author uses changes in county-level incarceration rates to make inferences about city-level crime rates.

Another point made by the CJCJ report is that “San Francisco reduced its incarceration rate (-38%) much faster than Sacramento County (-24%),” which he points to as evidence that more incarceration leads to more crime. But the incarceration rates he referenced are county-level, while the crime rates he presented above are city-level. In San Francisco this doesn’t matter because the city and county boundaries are the same. But in places like Sacramento where the city is only a portion of the entire county, you can’t use county-level data to make inferences about the city. Many people commit crimes within Sacramento county but outside of the city; these people would inflate the county’s incarcerated population but would not affect Sacramento’s crime statistics. However, the author mentions incarceration rates likely because it serves his narrative, and not because it adds to the quality of the analysis.

Problem #5: Any discussion of costs should be taken with a grain of salt, particularly when the author did not conduct a true cost-benefit analysis.  

The author also tosses in the fact that “Sacramento’s use of imprisonment cost California taxpayers $151.6 million, while San Francisco’s approach saved the state $163.3 million.” However, cost-benefit analyses are often very imprecise and can contain wide margins of error. There are multitude of things that one must consider when calculating costs and benefits (e.g., staff time and training, technology, lawsuits and appeals, and costs associated with recidivism), and it is impossible to account for them all. For the factors that can be accounted for, they are often not measured well, or are lumped into other categories of the city’s budget and remain undetermined. Thus, even well-done cost-benefit analyses are not as clear as one would think. These numbers presented by CJCJ sound great at the outset, but are very simplistic and probably tell us nothing.

Overall, this study was not methodologically sound enough to determine cause and effect. It also seemed like the author selectively shared results and threw in anecdotal details to craft a certain narrative, and not to improve methodological quality.