Propensity score matching works when the reasons people joined your program are things you actually measured before they joined. If enrollment was driven by motivation, referral judgment, or anything else you didn't record, matching will produce a tidy, balanced-looking comparison group that is still wrong.
That's the whole trick, and the whole catch. A propensity score is just each person's estimated probability of joining the program, given their baseline characteristics. Matching pairs participants with non-participants who had similar scores, so the two groups look alike on everything that went into the model. It is a good way to make a comparison fair on the variables you have. It cannot see the ones you don't.
When matching is a reasonable choice
- You know how people got in. If eligibility rules, staff referrals, or self-selection can be described in terms of recorded variables (prior test scores, income, risk level, age, neighborhood), matching can mimic the comparison those rules make unfair.
- You measured those things before the program started. Covariates have to come from intake or earlier. Matching on something the program could have changed, like attendance or a mid-program survey, quietly matches away part of the effect you're trying to find.
- You have a baseline version of the outcome. A prior score on the same measure is usually the single most valuable thing to match on. Without it, you're hoping demographics stand in for where people were already headed.
- There's a real pool of comparable non-participants. If nearly everyone with a high-risk profile enrolled, there's nobody to match them with. Say you have 300 participants but only 40 non-participants who look anything like them: you'll either throw most participants away or accept poor matches.
When it won't save you
- Selection on unmeasured traits. The classic case is a voluntary program. People who sign up for a job-readiness course are often more motivated than people who don't, and motivation rarely sits in an intake form. Two people with identical propensity scores can still differ in exactly the way that drives the outcome.
- A thin covariate list. Matching on age, gender, and zip code because that's what the database holds isn't adjustment, it's decoration. The method is only as good as the variables you feed it.
- Tiny samples. With a few dozen people per group, the propensity model itself is noisy and the matches get loose. Simple regression adjustment, stated honestly, may serve you better.
How to tell if it worked
Don't judge a matched analysis by its p-value. Judge it by three checks, all of which belong in the report:
- Balance. Compare the matched groups on every covariate using standardized mean differences, not significance tests. A common rule of thumb is that differences under about 0.1 standard deviations are acceptable.
- Overlap. Plot the propensity scores for both groups. If the distributions barely overlap, report who was dropped and admit the estimate only applies to the people who could be matched.
- Sensitivity. Ask how strong an unmeasured confounder would have to be to erase the effect. If a modest one would, say so in plain words.
It's also worth knowing the neighbors. Propensity score weighting keeps everyone instead of discarding the unmatched. And if you have outcome data from before and after the program for both groups, combining matching with a difference-in-differences comparison guards against stable differences that matching alone misses.
If you're deciding whether your enrollment data can support a matched comparison, or a funder has asked why your comparison group is fair, our program evaluation work starts by mapping how people actually got into the program, then picking the method that mapping can support.
Related reading