Flawed study oversells the benefits of Prop 47

A new report from the Center on Juvenile and Criminal Justice (CJCJ) presents some information on the impacts of Proposition 47 (Prop 47), calling it a “lifeline to California communities.” The report’s main conclusion is that Prop 47 is a success because it reduced prison costs without negatively impacting recidivism. The author contends that recidivism rates, homelessness, and unemployment all decreased after participating in Prop 47 programs, citing some data to support this conclusion. But a deeper dive into the source of these numbers proves quite skeptical, and whether or not these programs are actually effective remains unclear.

Further, the report also claims that Prop 47 has coincided with a period of record-low crime in California. Unfortunately, this statement is incredibly misleading. While overall crime rates might be down, this is largely driven by decreases in property crime, while some violent crimes (like murder) have been considerably high in recent years.

Through its reclassification of several low-level drug and property offenses, Prop 47 has reduced the state’s prison costs by $600 million. Per the report, $400 million of this was re-allocated to community-based programs that “address the root causes of crime and violence,” and the remaining $200 million was re-allocated to trauma recovery and school-based programs. The author of the report purports that these programs reduce recidivism, strengthen communities, and interrupt cycles of harm.

Prop 47 funds are distributed to grantees by the California Board of State and Community Corrections (BSCC) across overlapping four-year grant cycles. A total of 33 public agencies (e.g. behavioral health departments, school districts, etc.) received Prop 47 funding through Cohorts 1 and 2. By February 2022, the first cohort of grantees had completed their programs and provided final evaluation data on recidivism rates. The second cohort is halfway through its grant cycle and released initial evaluation data on housing and employment successes. Using this data, CJCJ examined impacts on homelessness, unemployment, and recidivism.

Homelessness and Unemployment

First, let’s look at homelessness outcomes. These outcomes are based on Cohort 2 participants only, as Cohort 1 did not report on these outcomes. According to the report, most Cohort 2 participants were unhoused when they enrolled in a Prop 47 program, but a “substantial share” had found housing by program completion. They claim that “homelessness fell by more than half, from 28% to 11%” and that “the share living independently nearly doubled, from 11% to 21%.” However, as mentioned in a discreet, carefully-worded footnote, these numbers reflect only 364 people–those who cited housing was a goal. This represents a small percentage of total Prop 47 participants (n=4,000+). But what about all the others who were enrolled in Prop 47 programs but didn’t seek housing assistance? The report says nothing about them or how they fared in the long run. This is problematic because those who actively sought housing services might be very different than those who did not, and it is quite possible that their success had to do with other factors (e.g., motivation) rather than Prop 47 itself.

Based on the percentages given, we can calculate the number of people who were homeless at the beginning versus the midpoint of the program. Of the 364 people who wanted housing, approximately 102 were homeless at the beginning of the program. 62 of these people found housing during their first two years in the program, while 40 remained homeless. Approximately 40 were living independently at the beginning of the program, which increased to 76 people by the two-year mark. In my opinion, the “great reductions in homelessness” can be better summarized as: Prop 47 helped 62 people find housing and helped 36 people live independently. Nothing is mentioned about the results of the entire cohort, and it is unclear how many individuals receiving Prop 47 services didn’t want help or weren’t looking to find housing.

Similarly, the report shows that most Cohort 2 participants were unemployed when they enrolled in a Prop 47 program, but a “substantial share” had found a job by program completion. According to the report, unemployment rates fell by one-third (64% to 42%) and rates of full-time employment more than doubled (17% to 39%). Again though, this key finding is based on only a subset of participants who wanted to find employment (n=628), which is a small percentage of the larger number of people affected by Prop 47 (n=4,000+). Thus, one has to wonder what happened to people who were eligible for programs under Prop 47, but didn’t want to work or utilize employment services. Again, we don’t know how the latter fared in the long run, which is problematic because individuals who actively seek employment are probably qualitatively different than those who do not. This means that impacts on employment may not have been due to Prop 47 itself, but might be due to other factors (e.g., motivation).

Based on the percentages given, we can calculate the number of people who were unemployed at the beginning versus the midpoint of the program. Of the 628 people who sought employment, 402 were unemployed at the beginning of the program. By the two-year mark, this had decreased to 264. In my opinion, the “large reductions in unemployment rates” can be better summarized as: Prop 47 services helped 138 people of the 4,000+ served find employment. Again, there is nothing else mentioned about the remainder of the cohort, i.e., those who received Prop 47 services but didn’t care to find employment. Thus, it seems crude to say that Prop 47 led to major decreases in unemployment, when employment outcomes were only examined for a small portion of Prop 47 offenders.

When it comes to housing and employment outcomes, the CJCJ report only examined 364 and 628 people, respectively. This sample seems shockingly small when you look at the total number of people who receive Prop 47 services. If you look here you will see that, around the program midpoint, Cohort 2 served 4,847 people in just one quarter (Quarter 8). In contrast, when the CJCJ report looks at people across the first eight quarters combined, their sample size is still nowhere near the size of the larger Prop 47-eligible population. For that reason, it’s impossible to say whether the sample of people examined here is representative of the full sample of people who received Prop 47 services. So, when the author boasts about huge reductions in unemployment and homelessness, one has to wonder what the outcomes are for the remaining 4,000+ people that receive some type of Prop 47 services.

Despite how convincing the CJCJ report numbers may seem, the reality is that the number of people actually impacted seems very small when compared with the larger number of people receiving Prop 47 services overall.

Recidivism

When it comes to recidivism outcomes, the CJCJ report relies on Cohort 1, as Cohort 2 has not yet reported on these outcomes. The author claims that “12 out of 13” grantees showed reductions in recidivism. This statement is misleading for a few reasons. First, a closer look reveals that there were actually 23 grantees in Cohort 1, not 13. However, only 13 provided sufficient data that allowed for comparisons between program participants and a “similar group who were not in the program.” But looking at the grantees’ evaluations more closely shows that most of the “comparison groups” weren’t true comparison groups. For this reason alone, many of the findings are invalid.

Despite the author’s claim that 13 grantees provided conviction data on a “similar comparison group,” this is simply not true. Five of the grantees (i.e., El Rancho School District, LA County Dept. of Health and Human Services, Plumas County, San Diego County, and San Joaquin County) provided no real comparison group, and simply compared their participants’ recidivism rates with more general recidivism rates of the county or state. Two grantees (i.e., Corning and San Francisco) relied on a pre-post design without a comparison group, and compared post-program criminal activity with pre-program criminal activity for the same people. Four grantees (i.e., LA City Attorney’s Office, Marin County Health and Human Services, Merced County Probation, and San Bernardino) used actual comparison groups, but did not ensure equivalence between groups. Further, the study from San Bernardino also compared different outcomes to each other, which was an additional flaw. Two grantees (i.e., Pasadena and Orange County) formed decent comparison groups that were statistically equivalent to each other, but the study in Pasadena also had incomparable outcomes from one group to the other, rendering it invalid as well.

After reviewing the evaluation reports for all of the 13 grantees, only one was highly rigorous (Orange County), and actually had a decent comparison group with valid outcomes that can be used to generate conclusions about the program’s impact. Following Orange County, there were three studies that met the minimum threshold of having an actual comparison group and a comparable outcome, and in these cases, the comparison group was still less than ideal (LA City Attorney’s Office, Marin County Health and Human Services, and Merced County Probation). Let’s go through the recidivism findings for each grantee that are shown on Figure 7 of the CJCJ report.

Corning

Project RESTORE in Corning was intended to help juveniles in the criminal justice system who have mental health or substance abuse issues. The program appeared to reduce recidivism rates, based on a pre-post design that examined changes in criminal behavior over time. After the program, 5% of youth recidivated, compared to pre-program rates of 25%. But unfortunately, this study did not actually utilize a true comparison group, and it only compares individuals’ prior recidivism rates before the program versus after the program. Single group studies are not strong enough to prove cause and effect, because there is no group of comparable people who did not receive services. It’s quite possible that people who participated in the program were qualitatively different than those who didn’t participate, and that their success is driven by other factors (such as personality type, social support) and not necessarily the Prop 47 program.

Pasadena

The “Vision 20/20” Reintegration Project (VTRP) in the city of Pasadena sought to address gaps in mental health and substance abuse services for formerly incarcerated individuals. The program seemed to reduce recidivism — apparently 17% of the treatment group recidivated compared with 31% of the comparison group. The comparison group in this study was actually good and the two groups looked very similar to each other in terms of demographics and other factors. Unfortunately, this study messed up on the outcome, and used different recidivism measures for each group. The 31% recidivism rate in the comparison group reflected arrests, convictions, and parole/probation violations, while the 17% recidivism rate in the treatment group recidivism reflected only new felony or misdemeanor convictions. The former casts a much wider net, so it is not surprising that the latter appears to be a “reduction.” Regardless, the two outcome measures aren’t comparable, rendering the findings invalid.

Los Angeles City Attorney’s Office

The Los Angeles Diversion, Outreach, and Opportunities for Recovery (LA DOOR) program intended to provide comprehensive, health-focused services for people at elevated risk of recidivating due to substance use, mental illness, or homelessness. The study conducted by the LA City Attorney’s Office was one of the better studies, and actually used a true comparison group. It showed that 15% of program participants recidivated compared with 25% of those in the comparison group, but this finding was not statistically significant. While the comparison group was mostly similar to the treatment group, the two were not entirely comparable. Most notably, the treatment group had higher numbers of prior convictions and warrants. Because of the baseline difference in prior criminal activity, it’s possible that the comparison group didn’t see as large of a reduction in recidivism simply because their baseline criminal activity was already somewhat lower.

Los Angeles County Health Services

The Reentry Intensive Case Management Services (RICMS) program in LA County delivered case management and navigation services to people who had been arrested, charged, or convicted of a crime and who had mental health and substance use issues. The study conducted in the LA County Department of Health Services saw re-conviction rates of 23% among program participants, compared with the overall re-conviction rate in LA County of 36%. Once again, this is not a true comparison group. This report mostly presents descriptive statistics on program participants, such as program enrollment numbers, client demographics, county healthcare utilization, and reconviction rates. It is unclear whether the people examined in this study were actually representative of the wider universe of offenders in LA County, so comparisons to the “typical” recidivism rate don’t mean much. Without a valid comparison group, it is hard to say whether reductions in recidivism were due to the program.

El Rancho Unified School District

The El Rancho Unified School District in the city of Pico Rivera utilized grant funds to implement a collaborative effort to reduce juvenile recidivism and delinquency. Only 7% of program participants recidivated compared with the “typical” juvenile recidivism rate of 50%. But the report does not use a real comparison group. Rather, they compare their juvenile participants’ recidivism rates with the typical recidivism rate for juveniles. It is unclear whether the juveniles examined in this study are actually representative of the wider universe of juvenile offenders, so comparisons to the “typical recidivism rate” don’t mean much. This method was insufficient to determine cause and effect, and it is unclear whether any impacts were actually related to the program.

Marin County

Marin County’s Improving Lives Via Opportunity and Treatment (PIVOT) program was a care coordination program for individuals with low-level criminal charges and behavioral health disorders. The study conducted by Marin County Health and Human Services showed recidivism rates of 9% for those who completed the program, compared with 23% of those who did not complete the program. Thankfully the study did use an actual comparison group, though it is not entirely comparable to the treatment group. All of the people in both groups had enrolled in a Prop 47 program and received some services, but those in the treatment group stuck with it while those in the comparison group did not. It is not surprising that those who completed the program had better recidivism outcomes — those who stuck with the program might be different from those who dropped out (e.g., they might have more motivation). Thus, it is hard to say that reductions in recidivism were actually due to the program itself, or whether they were due to other factors that differed between groups.

Merced County

The Los Banos Youth and Family Safety HUB in Merced County was designed to prevent youth recidivism through a comprehensive personal exploration and leadership development program. The study conducted by the Merced County Probation Department showed recidivism rates of 16% for those in the “HUB” program compared with rates of 35% that were seen in the comparison probation group. The study did utilize a true comparison group, which was good, though it was not equivalent to the treatment group. Groups differed in terms of recidivism risk factors at the outset, with people in the HUB reporting higher levels of gang participation, drug use, etc. than the larger probation group. This means that the HUB program and larger probation group may not equal in risk factors, which may affect program outcomes. Participants assigned to the HUB were assigned on a non-random basis, so another possibility is that people who were thought to succeed were hand-picked for the program. In the absence of randomization, it’s hard to know whether the reduced recidivism is truly a result of the program, or if individuals selected were particularly amenable to change. The total number of HUB participants in the analyses was relatively small, too (n=19), which limits the generalizability of program findings to other jurisdictions.

Orange County

In Orange County, Prop 47 funds were used to support intensive case management, linkages to treatment, and community supports for people in the criminal justice system with mental health or substance use disorders. The study by the Orange County Healthcare Agency found recidivism rates of 15% for people in the program, compared with 26% of those in the comparison group, a statistically significant finding. This study design was actually one of the better ones; it used a true, statistically-equivalent comparison group that was generated using propensity score matching. Researchers matched a group of people who received services after being released from jail (the treatment group) with a similar group who did not receive such services (the comparison group). The groups were statistically equivalent, but there was some attrition (the sample size decreased from 1,124 at baseline to 409 at the two-year follow-up). Thankfully, the authors mitigated the impact of attrition by applying statistical controls for various factors. The study found that people in the treatment group had significantly lower recidivism rates than their comparison counterparts, a statistically significant difference of 11 percentage points. One thing to note here though, is that the sample of offenders who were targeted for the Prop 47 intervention are likely not representative of the offending population as a whole. Individuals selected for the Prop 47 program had very specific needs related to mental health or substance abuse issues that have known treatment methods; therefore, it was presumed they might be more amenable to a Prop 47 intervention. For offenders without these specific needs, these types of services would likely not be as effective. While the program in Orange County seems effective in reducing recidivism, it’s important to keep in mind that the program was effective for very specific types of people with treatable mental health or substance abuse concerns.

Plumas County

The Plumas County District Attorney used their Prop 47 funding to implement pretrial diversion and pretrial release programs. Both programs provided collaborative services that were intended to help people address problems related to substance use, housing, work, and legal needs. Recidivism rates were 2% in the pretrial diversion program and 10% in the pretrial release program. These rates were compared with the overall recidivism rate of people under Pluma County probation, which was 67%. On its face, this looks like a huge reduction in recidivism, but similar to the other studies, the full universe of probationers isn’t necessarily a comparable group. We do not know whether those in the treatment group are similar to the wider net of probationers. There could have been something about the people selected for the treatment that made them more likely to succeed rather than just the program itself. For example, many of the people in the wider probationer pool also met the requirements but didn’t care to enroll, which could suggest that personal motivation to succeed plays a role in reducing recidivism rates.

San Bernardino County

The SAFE-T Net project supported by the San Bernardino County Department of Public Health attempted to combat recidivism by providing substance abuse and mental health services to formerly incarcerated people. Of the people referred to the program, 39% completed it. Of those completing the program, the recidivism rate was 15% (measured by arrests). When compared with the county’s overall recidivism rate of 58% (per a recent jail utilization study looking at jail bookings), the SAFE-T program appeared to reduce recidivism. At first this seems promising. However, the SAFE-T Net evaluation measured recidivism by subsequent arrests whereas the jail utilization study measured recidivism by subsequent jail bookings–two different measures of recidivism. The latter catches a wider net as it examined whether someone re-entered jail, even if it was only for a probation violation. Thus, it is not surprising that recidivism rates were larger for this group. Secondly, it is unclear whether the individuals referred to the program are representative of those in the larger universe of formerly incarcerated people, so it’s hard to say whether it makes sense to compare the two. For these reasons, the study is not ideal for being able to determine cause and effect.

San Diego County

In San Diego County, there were two programs, the Community Based Services and Recidivism Reduction (CoSRR) program and the San Diego Misdemeanants At-Risk Track (SMART) program. Unfortunately, the evaluation study was a pre-post quasi-experiment, and did not utilize a comparison group. They measured changes in recidivism over time (i.e., three-years prior compared with up to three years following the program) and compared people’s post-program criminal activity with their pre-program criminal activity. Another part of the report compared recidivism outcomes for people in the CoSRR and the SMART with numbers from a Public Policy Institute of California (PPIC) report that looked at recidivism rates pre- and post-Prop 47 in 12 California counties (as an aside, that report only found a reduction in recidivism of 3.1 percentage points following Prop 47 implementation). In the CoSRR and SMART programs, 57% and 67% of program participants recidivated (respectively), compared with the PPIC’s estimate of 46%. Based on these comparisons, it looks like Prop 47 programs actually increased recidivism. However, it’s still hard to know whether these findings are valid, mostly because it is unclear whether people in the Prop 47 programs looked similar to those people examined in the PPIC report.

San Francisco

The Promoting Recovery and Services for the Prevention of Recidivism (PRSPR) program implemented by the San Francisco Department of Public Health was designed to provide substance use treatment services for individuals with prior criminal offenses. The study evaluation unfortunately did not use a comparison group and rather, examined change from baseline to follow-up for program participants (i.e., a pre-post test). Initial recidivism outcomes did seem encouraging at first: baseline conviction rates were 4%, compared with post-program conviction rates of 2%. Out of the people examined, there were only six convictions in the six months prior to the program, compared with four convictions in the six months following the program. Although this shows a decrease in the number of convictions, the numbers were very small to start with and the change was not statistically significant. It is possible that people served by the program were not that likely to recidivate in the first place. In fact, for the vast majority of cases (80%) there was no change in convictions from baseline to follow up. Only four people (8%) saw an increase in the number of convictions after discharge. The authors also examined the successful program completers (n=16) separately, and still found no statistically significant reduction in convictions (pre- and post-numbers were still very low for this group). Regardless, the study did not use a comparison group, so even if findings were significant, we wouldn’t be able to say with certainty if it was related to the Prop 47 program.

San Joaquin County

The Homeward Bound program implemented by the San Joaquin County Behavioral Health Services aimed to deliver a variety of services (e.g. substance use counseling, psychotherapy, case management) to county residents, many of whom had prior criminal histories. For those in the program who had criminal histories, there was a re-conviction rate of 15.4%. When compared with the three-year reconviction rate in California of 46.5% (per the CDCR’s 2019 estimates), the program appears to have reduced recidivism. However, it’s not mentioned whether people participating in the program had the same characteristics as the wider universe of offenders in the state of California, so it’s challenging to make direct comparisons between these two groups.

Conclusion

The CJCJ report makes Prop 47 sound like a huge success in reducing recidivism, homelessness, and unemployment. Unfortunately though, it leaves out a lot of detail. For the homelessness and unemployment analyses, the report only examines a subset of offenders who received Prop 47 services. For the recidivism analysis, the report is missing key details about the actual studies that are summarized. Rather, it seems to regurgitate findings listed on the Prop 47 BSCC dashboard without going to the original sources for more information.

Upon looking at the details of each recidivism study, many of the methodologies are not sufficient to be able to ascertain cause and effect. One study (Orange County) out of the 13 is rigorous enough to be able to determine cause and effect. Importantly though, individuals had specific substance use and mental health needs with known treatments, so the results are probably not generalizable to the wider offending population. Regardless, this study found a 50% reduction in recidivism (26% to 15%, a difference of 11 percentage points) for program participants. The other 12 studies suffer from methodological errors that render their findings questionable. Before promoting Prop 47 programs as a huge success in reducing recidivism, more high quality studies should be conducted to see if they can replicate Orange County’s findings. Based on the research on Prop 47 programs, it’s not clear whether the cost savings have outweighed the potential safety risk that comes with reducing prison sentences.

Many people are afraid that Prop 47 may have increased recidivism by decreasing the severity of punishment associated with certain crimes. The CJCJ report argues otherwise, saying that Prop 47 programs actually decrease recidivism, thereby reducing costs without risking public safety. But based on the research cited by the CJCJ report, we cannot say for sure that recidivism has actually decreased as a result of Prop 47 programs.