In the spring of 1930, the Department of Health for Scotland ran one of the largest nutrition experiments ever attempted. Across 67 schools in Lanarkshire, some 20,000 children were enrolled. Ten thousand would get three-quarters of a pint of milk every school day for four months, half of them raw and half pasteurised. The other ten thousand would get nothing extra and serve as controls. Every child was weighed and measured in February and again in June. Gerald Leighton and Peter McKinlay wrote up the results, and the milk-fed children had grown more.
Then William Sealy Gosset, the Guinness brewer who published as "Student," read the report. In a 1931 paper in Biometrika he pointed out something awkward. At the start of the experiment, before a single bottle had been handed out, the control children were already taller and heavier than the children chosen for milk. On his reading, the gap amounted to about three months' growth in weight and four months' in height. The two groups weren't comparable on day one, so the comparison on the last day couldn't say much about milk. The cause, Student suspected, was kindness. Children were assigned by ballot or alphabetically, but head teachers were then allowed to swap children if the groups looked unbalanced. Faced with a thin, poorly fed child in the control group, plenty of teachers did what any decent person would do. They moved that child onto the milk.
Random assignment works for one reason only: nobody's judgment decides who gets the treatment. That's what makes the groups alike in everything, measured or not, so that a difference at the end can be credited to the treatment rather than to who was picked. The Lanarkshire design started out random and then invited people to improve on it. A single discretionary step was enough to undo the whole thing, because the teachers' choices weren't random. They were systematically tied to the very outcome being measured. Poorer children were smaller, poorer children were moved onto milk, so the milk group started smaller.
Notice that nobody cheated. Student was explicit that the bias was probably unconscious, driven by "the very human feeling" that the needier children deserved the milk more. That's what makes this kind of failure so durable. Good intentions don't announce themselves as bias, and no one reviewing the plan sees anything wrong with letting experienced teachers use their judgment.
Size didn't help either. Twenty thousand children sounds like it should swamp any small imbalance, but a big sample only shrinks random error. A systematic tilt in who gets treated survives at any scale. More children just give you a more precise estimate of the wrong thing. Student made the point memorably: he argued that 50 pairs of identical twins, one twin fed milk and the other not, would have given more reliable results than all 20,000 children. Well-matched comparisons beat huge, uneven ones. He also flagged smaller problems, like children being weighed in their clothes, heavier in February than in June, which muddied the weight measurements.
Nearly a century later, the statistician Stephen Senn reviewed the experiment and Student's critique in the European Journal of Epidemiology. His conclusion is blunt: adjusting for measured baseline differences can't repair a bias that came from how children were allocated within schools, and the experiment's value for understanding nutrition is doubtful. Student's points, he notes, still hold.
When someone tells you a study or pilot was randomized, ask one question: after the random assignment was made, could anyone change it? If the answer is yes, ask who, how often, and on what basis, and then check whether the groups still looked alike at baseline. A design that is random on paper but adjustable in practice is an observational study with extra paperwork. The fix is cheap if you build it in up front: lock the allocation, keep it out of the hands of the people who know the participants, and give staff a legitimate way to help the neediest children that doesn't run through the comparison group.
If you're planning a trial, pilot, or program evaluation and want an allocation process that survives well-meaning people, talk to us.
One email whenever we publish something new. No spam, unsubscribe anytime.
{{ subscribeErrorMsg }}