The Answer a Thousand PhDs Called Wrong

4 min read ▶ Listen to this post

On September 9, 1990, a reader named Craig F. Whitaker, of Columbia, Maryland, sent a probability puzzle to Marilyn vos Savant's "Ask Marilyn" column in Parade magazine. It was a version of an old game-show scenario: you're picking between three doors, one hides a car and two hide goats, you pick a door, and the host — who knows what's behind all three — opens a different door to reveal a goat. He then offers you the chance to switch to the last remaining door. Should you? Vos Savant's answer was unambiguous: switch. Staying with your original door wins the car one time in three; switching wins it two times in three. It's not a close call, and it's not a matter of opinion.

What happened next made national news. Within about a week, more than 10,000 readers had written in, the large majority telling her she was wrong — among them, by her own later count, close to a thousand with PhDs. Some letters were polite corrections; a number were not, including several from mathematics professors on university letterhead informing her, incorrectly, that she'd embarrassed herself in front of the country. The dispute got big enough that the New York Times put it on the front page the following July, and Monty Hall, the actual game-show host the puzzle was named after, ended up interviewed about it in his own living room. Even Paul Erdős, one of the most prolific mathematicians of the 20th century, refused to accept vos Savant's answer when a colleague explained it to him — he came around only after watching a computer run the game 100,000 times and tally the results. The math was never in doubt. Getting a room full of credentialed people to believe it was another matter entirely.

The mechanism: the host's move isn't random, and that changes everything

The reason this problem fools almost everyone the first time is that it looks like a coin flip that got easier: three doors, one is removed, two are left, so it must be 50/50. That reasoning would be correct if the door the host opened were chosen at random. It isn't. The host knows where the car is, and he is constrained to open a door that (a) you didn't pick and (b) has a goat behind it. That constraint means his choice carries information — it is never a neutral event, and treating it as one is exactly where the intuition goes wrong.

Work through it directly: your first pick has a 1-in-3 chance of being the car and a 2-in-3 chance of being a goat. If you picked a goat — the more likely case, two times out of three — the host has exactly one door left he's allowed to open, and it's forced to be the other goat, which means the remaining unopened door must be the car. Switching wins in that 2-in-3 scenario, every time. If you happened to pick the car on your first guess — the less likely case, one time out of three — switching loses. Add it up and switching wins two-thirds of the time, staying wins one-third. The two doors were never symmetric after the reveal, because the reveal itself wasn't a random draw. It was a move made by someone who already knew the answer and was limited in what he was allowed to show you.

Where else this shows up

• Academic and grant-funded research. When an interim look at accumulating trial data leads someone to drop a treatment arm, that arm's removal is informative, not random — it happened because of what the early data showed. Analyzing what's left as if it were still a clean, untouched comparison repeats the "must be 50/50 now" mistake with real consequences for a published effect size.
• Non-profit program evaluation. If a caseworker who already has a sense of the likely outcome decides which cases get referred for a second review, the pool that reaches the reviewer isn't a random slice of all cases — it's been shaped by someone who already knew something. Treating the reviewed cases as representative repeats the same error.
• AI and LLM evaluation. When a multiple-choice eval has an implausible option pre-filtered out by a step that already "knows" something about the correct answer — a rubric, an earlier model pass, a keyword filter — the remaining choices aren't a fresh, symmetric draw anymore. Scoring the filtered version as if it were the original question overstates how hard the remaining decision really was, and can make a model look sharper than it is.

What to ask for instead

Before updating your odds on anything based on an option, a case, or a data point disappearing from view, ask one question: was that removal random, or did it come from someone who already knew something about the answer? If it wasn't random, the naive updated probability is wrong, and the correct one usually isn't the intuitive one — it takes actually working through who chose what, and why, the way vos Savant's 10,000 correspondents mostly didn't bother to before writing in.

If you're working through a design or an analysis where a step in the process — a filter, a screening rule, an interim decision — might be quietly leaking information into what looks like a clean comparison, talk to us before you trust the odds at face value.

Related reading

• The Test Result That Fooled the Doctors — another case of credentialed experts badly misjudging a conditional probability under real time pressure, in a room instead of a mailbox.
• The Protective Effect That Wasn't There — a different way a correlation can be manufactured or distorted by the process that generated the data you're looking at, rather than by anything real underneath it.

Sources and further reading

• Tierney, J. (1991, July 21). Behind Monty Hall's Doors: Puzzle, Debate and Answer? The New York Times, p. 1.
• Kuhl, M. E. (2010). Monty Hall, Paul Erdős, and Monte Carlo. MWSUG 2010 Proceedings.

One email whenever we publish something new. No spam, unsubscribe anytime.