Lies, damn lies, and statistics

United States v. Johnson, a criminal case, identifies an erroneous statistics-based argument called “the prosecutor’s fallacy,” and the structure of that argument is of general interest to all litigators:

At Johnson’s second trial, the government introduced expert testimony about a partial DNA sample obtained from a bandana found in the vehicle used in the robbery. Testing yielded inclusionary match statistics capturing the probability that the sample was Johnson’s as compared to a coincidental match of an unrelated person, and the lowest inclusionary match statistic had an error rate of one in 4,100. That is, the expert explained, only one in 4,100 people would match the sample as strongly as Johnson did. But, in the government’s first closing argument, the prosecutor said that Johnson “left very little DNA, but he left just enough to prove that it was him in the front seat when you combine the 1 in 4,100 chance that it’s not him.” Johnson did not object.

 

The prosecutor’s fallacy occurs when “a juror is told the probability a member of the general population would share the same DNA is 1 in 10,000 (random match probability), and he takes that to mean there is only a 1 in 10,000 chance that someone other than the defendant is the source of the DNA found at the crime scene (source probability).” Conflating these two probabilities, as the prosecutor did here, yields “an erroneous statement that, based on a random match probability of 1 in 10,000, there is a 0.01% chance the defendant is innocent or a 99.99% chance the defendant is guilty.” 

No. 22-30421 (Oct. 26, 2023) (citation omitted). Note that the Court did not reverse on this issue in this case. The above graphic was supplied by DALL-E, a feature of the newest iteration of ChatGPT.