Part of our Statistical paradoxes series · Healthy user bias
In September 1991, Meir Stampfer and his colleagues published ten years of follow-up from the Nurses' Health Study in the New England Journal of Medicine. They had tracked 48,470 postmenopausal nurses, none with heart disease or cancer at the start, and compared the women taking estrogen with the women who weren't. The result was striking: after adjusting for age and other risk factors, current estrogen users had a relative risk of major coronary disease of 0.56. Roughly half the heart attacks. The authors even checked the obvious objection. Maybe estrogen users just saw doctors more often? Restricting to women who had recently seen a physician, the protective effect got slightly stronger, not weaker. Hormone therapy began to look like it could protect women's hearts as well as relieve hot flashes.
Then someone ran the experiment. The Women's Health Initiative randomized 16,608 postmenopausal women aged 50 to 79 to a combined estrogen-plus-progestin pill or a placebo, with heart disease as the primary outcome. On May 31, 2002, after a mean of 5.2 years out of a planned 8.5, the data and safety monitoring board stopped the trial. Breast cancer had crossed its stopping boundary, and the overall balance of harms and benefits had tipped the wrong way. Coronary heart disease, the thing hormones were supposed to prevent, came out with a hazard ratio of 1.29: higher in the treated group, not lower. In absolute terms that was about 7 extra heart events per 10,000 women per year, alongside 8 more strokes, 8 more blood clots in the lungs, and 8 more invasive breast cancers, offset by 6 fewer colorectal cancers and 5 fewer hip fractures. The pill hadn't changed. The comparison had.
In an observational study, nobody assigns the treatment. People choose it, or their doctors choose it for them. And the women who chose hormone therapy in the 1980s were not a random slice of postmenopausal women. Patients who start and stick with a preventive treatment tend to be more health-conscious in general: they exercise more, smoke less, have better access to care, and follow medical advice. All of that lowers heart risk on its own. Epidemiologists call this healthy user bias: the treated group looks healthier because healthier people are the ones who end up treated. Researchers were raising exactly this worry for HRT years before the trial. A 1996 paper in the American Journal of Epidemiology put it in its title: "Prior to use of estrogen replacement therapy, are users healthier than nonusers?"
The trap is that adjustment can't fully fix it. You can control for smoking, blood pressure, and weight because you measured them. You can't control for "the kind of person who takes her pills every day and gets her cholesterol checked," because nobody records that in a variable. That's why the physician-visit check in 1991 didn't help: it was testing one measurable piece of a mostly unmeasurable trait.
There was a second, subtler problem, and it's the one that eventually reconciled the two studies. The Nurses' Health Study compared current users, many of whom had been on hormones for years, against non-users. Anyone who'd had trouble early on, or stopped, was less likely to be in the "current user" group. In 2008, Miguel Hernán and colleagues reanalyzed the Nurses' Health Study data the way a trial would: comparing women who started hormone therapy with women who didn't, from the moment of that decision. Analyzed that way, the observational data showed a hazard ratio of 1.42 for the first two years and roughly no effect over the full follow-up. That's much closer to what the trial found. Same nurses, different question about when the clock starts.
It's worth being honest that the hormone story didn't end in 2002. The trial tested one combined pill in women aged 50 to 79, and researchers still argue about whether starting closer to menopause changes the picture. The methodological lesson isn't in dispute: an observational association between a chosen treatment and a good outcome is partly a portrait of who chose it.
Whenever someone shows you that people who used a treatment, program, or tool did better than people who didn't, ask: who decided who got it, and is there any reason the people who chose it were already headed for a better outcome? If the answer is "they chose it themselves," the honest options are a randomized comparison, a design that mimics one (comparing new starters with eligible non-starters from the same decision point), or a clearly stated caveat that the estimate includes selection. What isn't honest is a regression with a long list of covariates presented as if it settled the question.
If your study, evaluation, or product analysis compares people who opted into something with people who didn't, talk to us before healthy users write your headline for you.
One email whenever we publish something new. No spam, unsubscribe anytime.
{{ subscribeErrorMsg }}