When Does a Program Need a Control Group?

3 min read ▶ Listen to this post

A program needs a control group the moment you want to claim it caused an outcome, not just that the outcome happened. If you're reporting how many people enrolled, how satisfied they were, or how a number moved between intake and exit, you don't need one yet — that's a description of what occurred. The moment "because" enters the sentence — participants improved because of the program — you need something to compare against, because almost everything else that could produce the same before/after number is still sitting there unaccounted for.

What a control group actually rules out

A comparison group isn't bureaucratic overhead; it's the only way to separate the program's effect from everything else that changes a number over time. Without one, at least four explanations are competing with "the program worked" for the same data: participants who were doing unusually badly when they enrolled tend to improve somewhat on their own (regression to the mean); people mature, learn, or recover regardless of intervention; broader trends — a better job market, a new policy, a seasonal effect — move the outcome for everyone, participants or not; and the people who sign up for a voluntary program are usually not a random slice of the population, so their trajectory was never going to match anyone else's. A control group — a genuinely comparable group that didn't get the program — is what lets you subtract those explanations out and see what's left.

The clearest signs you need one

• You're using the word "caused" or "because." Any claim that the program produced the outcome, rather than merely preceded it, needs a counterfactual to be credible.
• A funder or grant renewal is asking for evidence of impact, not just activity. "We served 400 families" answers a different question than "the 400 families we served ended up better off than they would have otherwise" — and funders increasingly know the difference.
• Participants opted in. Self-selected groups differ systematically from non-participants in ways that are hard to fully measure or control for after the fact — motivation, prior severity, access, awareness.
• The outcome you're tracking moves on its own anyway. If the metric naturally drifts over a year — test scores, recidivism, health measures — a before/after change could just be that drift, with the program contributing nothing.

When a true control group isn't possible

Randomizing who gets a program is often the cleanest design and sometimes the right call, but it isn't always ethical or feasible — you can't randomly deny a service people are entitled to, or a waitlist may not exist. In those cases, a weaker but still genuine comparison is usually available: a staggered or phased rollout that lets early cohorts serve as a comparison for later ones, a matched comparison group built from people who look similar on the variables that predict the outcome, a difference-in-differences design comparing the change in participants against the change in a similar untreated group over the same period, or a regression discontinuity design when eligibility is decided by a cutoff score or threshold. None of these are as clean as a randomized trial, but all of them beat a single before/after number, because they at least attempt to answer what would have happened anyway.

What this costs you if you skip it

The risk isn't just a weaker report — it's a specific one. Programs that look effective on a before/after number and then get evaluated with a real comparison group have, more than once, turned out to have no effect or a negative one, after everyone involved was already convinced otherwise. That's a harder conversation to have after three years of reporting the wrong number than before you started.

If you're scoping a program evaluation and aren't sure what kind of comparison group is realistic given your constraints, that's exactly the design question we help sort out early, before the data collection locks you into an answer you can't fully trust.

Related reading

• The Program That Looked Like It Worked — Until Someone Checked — a real program that felt effective on every before/after signal available, until a genuine control group showed the opposite.
• What Funders Actually Expect From a Program Evaluation — why funders are increasingly asking for causal evidence, and a control group is usually what that evidence requires.

One email whenever we publish something new. No spam, unsubscribe anytime.