There's no single safe number. A study that loses 30% of its participants evenly across groups, for reasons unrelated to the outcome, can still give you a trustworthy answer. One that loses 10%, mostly from the comparison group or mostly the people who were struggling, may not. What decides it is who left, not just how many.
Overall attrition vs. differential attrition
Overall attrition is the share of everyone who enrolled that you have no outcome data for. It mainly costs you precision: fewer people, wider confidence intervals, less power to detect an effect. That hurts, but it's honest.
Differential attrition is the gap in dropout rates between the program group and the comparison group, and that's the one that biases the answer. Say 85% of program participants complete the follow-up survey but only 65% of the comparison group do. The comparison members you still reach are the ones with stable housing, the same phone number, and enough interest to answer, and they were probably going to do better anyway. The program's effect is now distorted, and a bigger sample won't fix it. Evidence reviewers know this. The federal What Works Clearinghouse, for example, rates studies on overall and differential attrition together, and the more overall attrition a study has, the less differential attrition it tolerates.
The questions that actually decide it
- Is the dropout rate similar across groups? A gap of a few percentage points is usually tolerable. A gap of 15 or 20 is a red flag whatever the overall rate.
- Do the people who left look different at baseline? Compare dropouts and completers on intake measures like prior scores, risk level, and demographics. If dropouts started out worse off, the completers will flatter whichever group held on to more of them.
- Is leaving related to the outcome itself? In a job-training program, the people you can't reach six months later might be the ones who found work and moved, or the ones who didn't and gave up. Those two stories push the estimate in opposite directions, so work out which one is plausible before you interpret anything.
- Is it missing data, or just missing participation? Someone who stopped attending but still answered the follow-up survey hasn't left the evaluation. Keep them in the group they were assigned to. That intent-to-treat comparison is what funders and reviewers expect.
What to do about it
- Prevent it in the design. Collect several ways to reach each person at intake, budget for follow-up incentives, and make follow-up exactly as easy for the comparison group as for participants. Most differential attrition comes from chasing program participants harder than everyone else.
- Use a principled missing-data method. Multiple imputation or full information maximum likelihood, using baseline variables that predict both dropout and the outcome, beats analyzing completers only. Both rest on assumptions too, but at least the assumptions are stated.
- Run a sensitivity analysis. Ask how bad the missing participants' outcomes would have to be to erase the effect. If a mild assumption wipes it out, say so. If it takes an extreme one, your finding just got stronger.
- Report it plainly. A flow diagram showing how many people were enrolled, assigned, followed up, and analyzed in each group, plus a baseline comparison of dropouts and completers, answers the reviewer's first question before they ask it.
If you're planning a follow-up survey, or already sitting on a dataset with more empty rows than you'd like, our program evaluation work starts with exactly this question: what can the data you have still honestly support?
Related reading