The “Make-it-Right” (MIR) program is a restorative justice conferencing and diversion program that was implemented in San Francisco for high-risk teenagers facing medium-severity felony offenses (e.g., burglary, assault, motor vehicle theft). The National Bureau of Economics Research (NBER) recently published a working paper that boasts a 30% reduction in four-year recidivism rates for youths in the program when compared with a control group. The researchers claim that the study is especially strong due to being a randomized controlled trial (RCT), i.e., the strongest type of individual research study. However, upon closer review of the study, there is reason to be skeptical of the results.
To summarize, it looks like the randomization method was severely compromised, rendering the “key strength” of the design effectively invalid. Now, this does happen in research sometimes, and there are ways to try to deal with it. However, I am disappointed that the authors did not acknowledge this problem, nor did they take any steps to mitigate it. Below, I provide a review of the study, explain how the randomization went wrong, how this affects the results, and some steps that the authors should have taken (but didn’t).
Continue reading . . .