The Dead Salmon That Read Minds

3 min read ▶ Listen to this post

In 2009, a neuroscience grad student named Craig Bennett bought a whole Atlantic salmon from a fish market, put it in an fMRI scanner, and showed it a series of photographs of people in social situations. Through a mock task, the salmon was asked to judge what emotion each person in the photo was feeling. The fish, as far as anyone can tell, felt nothing about this. It was dead. It had been dead since before it arrived at the lab.

When Bennett ran the scan data through the same statistical pipeline used in a typical published fMRI study, he found it: a cluster of voxels in the dead salmon's brain cavity lighting up with what the software called statistically significant activity, tracking with the emotional content of the photos. Nothing was happening in a dead fish's brain. What was happening was that the analysis had tested roughly 8,000 tiny brain regions independently, at a significance threshold built for testing one thing at a time — and out of 8,000 coin flips, some of them come up "significant" purely by chance. Bennett and his colleagues wrote it up as "Neural Correlates of Interspecies Perspective Taking in the Post-Mortem Atlantic Salmon," presented it as a conference poster, and it went on to win an Ig Nobel Prize. The joke landed because the underlying math was completely real, and a lot of published neuroscience at the time wasn't correcting for it.

The mechanism: run enough tests, and chance starts looking like a finding

A p-value of 0.05 means there's a 5% chance of seeing a result at least that strong if nothing real is going on. Run one test, and 5% odds are reasonably safe. Run 8,000 tests — one per brain voxel, one per subgroup, one per outcome metric — and you'd expect around 400 of them to clear that bar on pure noise alone, even if the true effect everywhere is exactly zero. Nobody has to do anything dishonest for this to happen. The dead salmon's "activity" wasn't fraud; it was what a normal, correctly-run statistical test looks like when it's asked the same question thousands of times without any adjustment for how many chances it's had to get a false positive.

This is the multiple comparisons problem, and its quieter cousin is the garden of forking paths: even a single dataset, analyzed one outcome at a time, offers a researcher dozens of small decisions along the way — which covariates to control for, which subgroup to break out, which outlier to drop, which of three plausible outcome measures to lead with. Each decision is defensible in isolation. Made after peeking at what makes the result look best, a handful of small, reasonable-looking choices can manufacture a "significant" finding out of nothing, the same way testing 8,000 voxels can light one up in a fish that's been dead for a week.

Where else this shows up

• Academic and grant-funded research. A study that tests a dozen subgroups — by age, by dosage, by site, by baseline severity — and reports only the one or two that hit significance is running the same experiment the salmon ran, just with people instead of voxels. The abstract shows the hit; the eleven misses usually don't make it into the paper at all.
• Non-profit program evaluation. A program that tracks fifteen outcome metrics and highlights the three that improved is doing informal multiple comparisons without a correction. Fifteen metrics, tested loosely, will usually turn up a few that moved in the right direction by chance alone — regardless of whether the program did anything.
• AI and LLM evaluation. Running a model against many benchmark slices, prompt variants, or scoring rubrics and reporting the configuration that beat the baseline is the same pattern in a newer wrapper. With enough slices tested, one of them clears the bar on noise, and it's tempting to call that one the headline result.

What to ask for instead

Before trusting a "significant" result, ask a version of one question: how many things were tested to get this one, and was the threshold for "significant" adjusted for that? A single pre-specified test at p < 0.05 and the best of forty exploratory ones at p < 0.05 are not the same claim, even though they can look identical in a results table. The fix isn't complicated — correction methods for exactly this problem have existed for decades — but it only gets applied when someone asks whether it was needed.

If you're looking at a result that survived one test out of many, or designing an analysis where that risk is baked in from the start, talk to us before you write it up as a finding.

One email whenever we publish something new. No spam, unsubscribe anytime.