In January 2024, a team at UC San Diego Health published a study of COMPOSER, a deep-learning model that watches the records of emergency department patients and raises an alert, delivered to nurses, when it predicts sepsis. Sepsis is the body's extreme response to an infection, and early treatment matters. The study covered 6,217 adult sepsis patients across two emergency departments, from January 2021 to April 2023. The headline in the press release and the coverage was a 17% reduction in mortality.
That number is accurate. It is also one of two ways to describe the same result, and the two sound very different. In the paper's own terms, the model's deployment was associated with a 1.9 percentage point absolute reduction in in-hospital sepsis mortality, which is the 17% relative reduction. The two figures are linked by a baseline. If a drop of 1.9 points is 17% of the old rate, the old rate was about 11%, the new rate about 9%, and the difference is the 1.9 points.
Picture 1,000 sepsis patients. At the earlier rate of about 11%, roughly 112 die. At the later rate, roughly 93 die. That is 19 fewer deaths per 1,000 patients. It is also 17% fewer deaths, and it means about 53 patients needed to be treated under the new system for one extra person to survive. These are derived from the study's two published numbers and are only approximations. All three descriptions are true. They land differently. "A 17% reduction" sounds like a breakthrough, "19 fewer deaths per 1,000" sounds meaningful and finite, and "53 patients per life" sounds like a decision about cost and effort.
Relative risk is not wrong, but it cannot be interpreted without the baseline. The same 17% reduction applied to a condition with a 1% mortality rate would save 1.7 lives in 1,000, not 19. When a result is reported only in relative terms, the reader cannot tell whether the stakes are large or small.
The study was a before-and-after comparison: outcomes in the 705 days before the alerts were compared with outcomes in the 145 days after. There was no group of patients treated at the same time without the alert. The authors describe the design as quasi-experimental and report the result as an association. In a before-and-after design, anything else that changed between the two periods, such as treatment practices, patient mix, staffing or seasonal infection patterns, can look like an effect of the tool. The authors also reported higher compliance with the sepsis care bundle after deployment, which suggests a plausible route to the benefit, but it does not remove the design's limits. Several of the authors are co-founders of a company that commercializes decision-support tools like this one, a disclosure the paper makes and one worth noting when weighing any evaluation done by the developers themselves.
None of this says the tool does not work. It says that a single before-and-after comparison, with a headline expressed in relative terms, is a starting point for a stronger evaluation, not the end of one.
For any claim of improvement, ask for the baseline rate, the new rate, and the absolute difference. Ask how many people it applies to, and whether there was a comparison group over the same period. If the number is the only thing you can find, treat it as a question rather than an answer.
If you are reading an evaluation, or planning one, and need to know what the headline number is really saying, send it over. Working out what it supports is usually a short conversation.
One email whenever we publish something new. No spam, unsubscribe anytime.
{{ subscribeErrorMsg }}