Part of our Statistical paradoxes series · Lord's paradox
In 1967 the psychometrician Frederic Lord published a two-page note in Psychological Bulletin built around a deliberately simple scenario. A university dietitian wants to know whether the food in the dining halls affects students' weight. Every student, men and women, is weighed in September and again in June. There is no control group, because everyone eats in the same halls.
Two statisticians get the same data. The first looks at each group's weight at the start and end of the year. For the men, the average weight is the same in June as in September. For the women, the same. Nobody's group changed, so the first statistician reports that the dining halls had no effect on either sex. The second statistician asks a different question: among students who started at the same weight, who ended up heavier? Adjusting for September weight with an analysis of covariance, the answer is the men. At any given starting weight, men finished the year heavier than women, so the second statistician reports that the dining halls affected the sexes differently.
Both analyses are standard. Both were done correctly. They cannot both be the answer to "what did the dining hall do?"
The two answers diverge when the groups start in different places, and men and women do differ in average weight. That gap between the groups is what makes the choice of analysis matter.
The first statistician is comparing each group to itself. Her implicit assumption is that, without the dining hall, a student's June weight would have equaled their September weight. The second statistician is comparing students across groups at the same starting weight. His implicit assumption is that June weight depends on September weight in the same way for both sexes, apart from a constant difference between them. Within a group, students who start unusually heavy tend to finish closer to their group's average, which is ordinary regression to the mean. A heavy woman and a light man can share a September weight, but each is heading back toward a different group average, and that pull is what the covariate adjustment turns into an apparent "effect."
Neither assumption can be tested with this data. With no control group, nothing in the data says what a student's weight would have been without the dining hall. That is the statisticians' real disagreement, and it is about causation, not arithmetic.
Paul Holland and Donald Rubin, in a paper written for a volume honoring Lord, argued that the two statisticians' descriptive conclusions are both correct: they just describe different aspects of the data. Their causal conclusions, they showed, rest on different untestable assumptions about the missing comparison, so neither is simply right or wrong. Judea Pearl later returned to the puzzle using causal diagrams, treating it as a question about whether adjusting for the baseline is appropriate, which depends on the causal story you believe and not on anything in the table.
The lesson Lord drew is the one that holds up: when groups differ before the intervention, no statistical procedure can be counted on to make proper allowances for those preexisting differences. The method does not rescue the design.
Any evaluation that compares change in groups that did not start alike runs into this:
Do not ask which statistic is correct. Ask what the comparison group would have done without the intervention, and whether the data can say. If you have a genuine comparison group measured over the same period, a difference-in-differences design states its assumption openly: that the groups would have moved in parallel. If you do not, report both analyses, say plainly what each assumes, and treat the disagreement between them as information about how fragile the conclusion is.
If two reasonable analyses of your before-and-after data are pointing in different directions, send us both. Working out which assumption your study can defend is usually a short conversation.
One email whenever we publish something new. No spam, unsubscribe anytime.
{{ subscribeErrorMsg }}