San Francisco risk assessment tool: Efficacy in predicting public safety risk for pretrial release decisions

A recently published San Francisco-based study conducted by the California Policy Lab at UC Berkeley/Los Angeles has shed some insight on the accuracy of the city’s Public Safety Assessment (PSA), an algorithmic tool that is used to inform pretrial release decisions for adult offenders. The tool scores defendants on how likely they are to show up for future court dates, their probability of committing a crime during the pretrial phase, and whether that crime might be violent. Overall, the researchers concluded that the tool met the threshold required for it to be considered “sufficiently predictive” (p. 27) of risk. The study examined 9,800 individuals released pending trial between May 2016 (when the tool was adopted) and December 2019. Of those people released, 51% failed to appear in court and 55% were arrested for new crimes during the pretrial release period (18% of which were for violent offenses).

The use of risk assessment tools, such as the PSA, has increased in recent years in California and in the U.S. more broadly. As such, many have called for validation of their accuracy and reliability in predicting public safety risk. In October 2019, California Senate Bill 36 was passed, requiring that each pretrial services agency using one of these tools (e.g., the PSA) conduct studies validating their accuracy by July 1, 2021 and every three years thereafter.

The PSA was originally developed by Arnold Ventures (previously the Laura and John Arnold Foundation) using a nationally-representative sample of 750,000 cases from nearly 300 U.S. jurisdictions. It has since been adopted in several counties throughout the country, including several in California. The tool is based on nine risk factors (prior convictions, incarceration history, failures to appear, violent offenses, pending cases at the time of arrest, and age) that are used to predict failures to appear, new criminal arrests, and new violent criminal arrests. The former two outcomes are predicted using a numeric scale while the latter is predicted using a binary indicator. The risk scores and accompanying recommendations do not replace judicial discretion, but are used to guide their decision as to whether to release or detain a defendant. In San Francisco, judges complied with the recommendations given by the tool about 70% of the time.

The San Francisco study found that all three outcomes were more likely to occur during the pretrial release period as risk score(s) increased, indicating that the tool was fairly accurate in its predictions. Of the individuals released, 51% failed to appear in court, 55% were accused of committing a new crime, and 18% were accused of committing a new violent crime.  Of those released, people in the lowest risk category were most likely to attend their court hearings (66%), compared with those in the two highest risk categories (30-35%). Similarly, rates for new criminal arrests were higher for those in the highest risk category (74%) compared with those in the lowest risk category (33%). Regarding new violent offenses, individuals with the binary violent risk indicator had 20% great odds of committing a new violent offense during the pretrial period than those without the indicator. The study also examined disparate effects in the tool’s predictions based on race, finding that Blacks were more likely to be detained than Whites or Hispanic individuals. However, when the PSA risk score was controlled for, this relationship disappeared.

Following the release of these findings, San Francisco Supervisor Catherine Stefani introduced new legislation that would require more extensive reporting of court appearances and re-offense rates. The ‘extensive reporting’ would extend the duration of reporting throughout the entire pre-trial release period, expand reporting of statewide offenses in addition to just local offenses, and include breakdowns by PSA score and release type.

The San Francisco study follows results from a Los Angeles County study (released in September 2020) conducted by the Center for Evidence-Based Corrections at UC Irvine. In Los Angeles, the PSA was also considered sufficiently predictive, with two of the three measures (failure to appear and any new criminal arrest) being considered “strongly” predictive. The sample in this study included over 720,000 people released pretrial between January 2015 (when the tool was adopted) and December 2018. Of the people released, 56.2% failed to appear to court, 47.9% were arrested for a new crime, and 9.8% were arrested for a new violent crime. People released on bail or bond were most likely to attend court appearances (65% and 63.5%, respectively) in comparison with those released on their own recognizance (51.5%) or citation-only (31.1%). In terms of being arrested for a new crime, citation-only releases posed the greatest risk, with 56.2% being arrested for a new crime during the pretrial period. In contrast, re-arrest rates for any new crime were approximately 37% in the other categories, ranging from 33.3% in the bail group, to 37.3% in the released on recognizance group, and 39.9% in the bond group. People released on citation-only were most likely to be re-arrested for a new violent crime (10.5%), compared with those released on bond (9.8%), those released on their own recognizance (8.2%), and those released on bail (8.0%).

One limitation of many validation studies, though, is that outcomes used in the analysis are observed only for individuals who are actually released and obviously cannot be observed for those who are detained. Thus, as seen in San Francisco, the detained population might look substantially different than the released population, which could have affected results. Further, it is important to keep in mind that pretrial release and bail reforms continue to evolve over time, which could affect patterns of public safety risk. It will be important to remain updated on the research in this area, and we should expect more studies in the future given the conditions set forth in Senate Bill 36.