Red/Blue States, Crime Rates, and Statistical Prestidigitation

In the past two years, there have been dueling studies flying back and forth about whether jurisdictions in the political control of one party or the other have higher crime rates. They have been used by advocates to make “studies show” arguments in favor of or against particular policies. But this is all smoke and mirrors, as this Issue Brief from the Manhattan Institute shows. It is titled The “Red” vs. “Blue” Crime Debate and the Limits of Empirical Social Science, by George J. Borjas and Robert VerBruggen.

This is really a case study in how researcher-advocates can produce any bottom line they want in many cases through design decisions that fly beneath the radar of public awareness. Do you compare states or counties? What variables do you control for? The authors note, “Casual consumers of empirical social science research often fail to appreciate all the ways in which researchers can manipulate the data to say whatever they want.” An alternate expression is the pithy old saying, “Figures don’t lie, but liars figure.”

Failure to control for relevant variables has long been a problem in crime studies. It is a major reason why such studies are so difficult, as there are a great many variables. One particularly notorious example was a study in the 1980s that went all the way to the Supreme Court. David Baldus et al. claimed that their study showed that defendants who killed white victims were more likely to be sentenced to death than those who killed black victims. The Federal District Court held that the model showing that result “is totally invalid for it contains no variable for strength of the evidence, a factor which has universally been accepted as one which plays a large part in influencing decisions by prosecutors.” See this article. Yet this finding is nearly forgotten, and the study in question is regularly cited as if it were valid.

What happens to the red/blue divergence if relevant variables are controlled? Borjas and VerBruggen give it a shot by controlling “for the shares of the population that are male, black, Hispanic, and Asian; for the percentage of the population that is urbanized; for the share of the population that is aged 15–24 and 25–34, as well as black males in those age ranges; and for per-capita income.” With these variables controlled, the divergences that supposedly favor “red” counties or “blue” states (take your pick) disappear into the statistical grass.

Is this the definitive word? No. Others may criticize the way this study was done, and on it goes. But that is not the main point. This is:

It seems to us that it would be far more productive to spend that time and effort debating the merits of actual policies, as opposed to measuring the effect of partisan leanings in the population. Democrats say that lax Republican gun laws drive up murder; Republicans say that Democratic mishandling of policing and prosecution is what really matters. Though our cross-sectional data are not suited to studying these hypotheses—for one thing, police staffing and gun ownership can change in response to crime, in addition to whatever effect they have on crime—there are large and important academic literatures on both topics.

“Cross-sectional data” is study-speak for data gathered at one snapshot of time. Better information on cause-and-effect relationships can be gathered from studies across time. If A and B tend to go together but the rise in B follows the rise in A, we can rule out the possibility that B causes A, rather than the other way around. But there is still the possibility that both are caused by C, which is why the choice of control variables will always be critical.

One big problem with long-term studies is that present analyses are limited by decisions made years ago on what data to gather. There is an old joke among landscapers: “When is the best time to plant a tree? Twenty years ago.” That is also a good time to start a longitudinal study.

Meanwhile, be skeptical about simplistic bottom lines of studies. What “studies show” “ain’t necessarily so.”