Policy-Based Evidence-Making

Steven Hayward at Powerline Blog has this post on appeals to expertise in policy arguments. The specific examples he discusses are off-topic for this blog, but the general point is an issue that comes up often in criminal justice debates. The post includes a particularly nice turn of phrase that I expect to quote often.

Whenever a progressive says we should “follow the evidence” because we must have “evidence-based policy-making,” you should reach for your wallet (for starters). Because today we all too often have the opposite: policy-based evidence-making.

In criminal justice, you don’t need to reach for your wallet. You need to run for your life or reach for your gun.

Appeals to experts often refer to studies that began with a desired policy goal and then designed a study to create ammunition to further the goal. To illustrate, I will briefly review one area of research/advocacy that I have examined in some depth — the claim that the death penalty is administered in a racially biased manner. For the long form and the citations, see this article.

The earliest and best-known study in the field was commissioned by the NAACP Legal Defense and Education Fund. By the 1980s, that organization had strayed far from its original mission of fighting for justice and equality and had instead adopted a major policy goal of getting murderers off with less than their deserved punishment, i.e., opposing justice.

The study gathered data in Georgia murder cases before and after the reforms that followed Furman v. Georgia (1972). The researchers attempted (and I emphasize attempted) to control for all the legitimate factors that would determine whether a murderer received a death or life sentence to see if the illegitimate factor of race played a detectable role.

The first part of the analysis examined the form of discrimination that had always been the greatest concern and that actually lay behind the Furman decision — the possibility that black murderers were sentenced to death for crimes that would have drawn a life sentence for white murderers.

The bombshell result was negative. While such discrimination was discernible in the pre-Furman cases, it was not in the post-Furman cases. The reforms had worked. “What is most striking about these results is the total absence of any race-of-defendant effect,” the researchers reported. If NAACP-LDEF had really been interested in equal treatment, this result would have been cause for celebration and shouted from the rooftops. Instead, they emphasized the result on a different and less compelling subspecies of discrimination.

They claimed the data showed discrimination based on the race of the victim. But actually it did not. The researchers had several models, and they cherry-picked the one that showed the desired result to make their case. That model omitted the important and legitimate factor of the strength of the prosecution’s case for guilt. When guilt is less than certain, prosecutors are quite legitimately less inclined to seek the death penalty, and jurors are quite legitimately less inclined to impose it.

The federal district judge saw right through this manipulation. “The best models which Baldus was able to devise which account to any significant degree for the major non-racial variables, including strength of the evidence, produce no statistically significant evidence that race plays a part in either [the prosecutor’s or the jury’s] decisions in the State of Georgia.” McCleskey v. Zant, 580 F. Supp. 338, 368 (N.D. Ga. 1984) (emphasis omitted).

Yet hardly anyone knows this important result, even people who work in the field. The expert/advocates’ spin on the data is far better known than the judge’s findings of fact. The studies claiming to find discrimination–all of which, to my knowledge, were conducted by people opposed to the death penalty–are regularly cited as fact. Those not finding discrimination are rarely mentioned in the press and largely forgotten.

This story contains many of the elements of misuse of research: advocacy motivation, cherry-picking results, model selection, and biased reporting. Fortunately, the Supreme Court didn’t buy it back then.

In more recent years, we have seen expertise misused in criminal cases in the Supreme Court. The Court’s headlong rush to let under-18 violent criminals off with less than they deserve has been regularly supported (using that term loosely) with appeals to a body of science that does not really support the rules of law being announced. See our brief in Graham v. Florida, part III.

When it comes to deciding which murderers are intellectually disabled, and hence exempt from capital punishment, the Court has accepted assertions of authority from organizations that change their standards without any stated basis in scientific research. These are partly political organizations with motives to expand government benefits, expand insurance coverage of treatment, and reduce punishment. See our briefs in Hall v. Florida and Moore v. Texas. Yet the Court accepted and repeated their spin because, frankly, a majority of Justices desired the result.

Perhaps in the current virus crisis large numbers of people will become aware that mathematical models vary all over the place, and many of the modelers have policy motivations. That may lead to greater skepticism overall, a desirable result. As Steven says, “It is not ‘anti-science’ to be skeptical of claims to expertise in social and political matters.”