Sometimes. You need a correction when several tests share one claim — when any one of them coming up "significant" would let you say the study found something. You usually don't need one when each test answers its own separate, pre-specified question. The hard part is deciding which situation you're actually in, and deciding it before you see the results.
Why it matters at all
At a 0.05 threshold, each test of a true null has a 5% chance of a false positive. Those chances add up fast. Say you compare a program and control group on 10 independent outcomes where the program truly does nothing: the chance that at least one comes up significant is about 40%. Report whichever one did, and you've published noise with a p-value attached. A correction keeps the error rate for the whole set of tests, not each test alone, near the level you meant.
When you probably need one
- Several outcomes, one headline. If the conclusion will be "the program worked" when any of five outcomes improves, those five tests are one claim, and the threshold should reflect that.
- All pairwise comparisons. Four groups give six pairwise tests. Post-hoc comparisons after an ANOVA are the textbook case for adjustment (Tukey, Holm, or similar).
- Subgroup hunting. Breaking results out by site, age band, gender, and baseline severity multiplies the tests quickly. Unadjusted subgroup "findings" are where many false positives come from.
- Screening many variables. Gene expression, item-level survey analysis, dozens of predictors tested one at a time: here false-discovery-rate methods (Benjamini–Hochberg) are usually the better fit than strict family-wise control.
When you usually don't
- One pre-specified primary outcome. If your analysis plan names a single primary test and everything else is labeled secondary or exploratory, the primary result stands at its nominal threshold.
- Genuinely separate questions. A study asking whether tutoring raises reading scores and, separately, whether it changes attendance can treat those as two claims, as long as neither one is presented as a backup for the other.
- Descriptive or exploratory work. If you're reporting estimates and intervals to generate hypotheses, say so plainly and don't present any single p-value as a confirmed finding.
Picking a method without overthinking it
- Bonferroni divides the threshold by the number of tests. Simple and defensible, but conservative when tests are correlated or numerous.
- Holm controls the same error rate as Bonferroni and is never less powerful. There's rarely a reason to prefer plain Bonferroni over it.
- Benjamini–Hochberg controls the expected share of false discoveries rather than the chance of any. It suits screening, where some false leads are acceptable.
- Hierarchical (gatekeeping) testing tests outcomes in a pre-set order and moves on only when the previous one succeeds. It's common in trials with a primary and key secondary outcomes, and it spends no extra alpha.
Whatever you choose, the choice belongs in your statistical analysis plan, along with which tests form each family. A correction picked after seeing which one makes the result survive is just another forking path. And report the number of tests you ran, not only the ones that cleared the bar. Reviewers and funders increasingly ask.
If you're writing an analysis plan for a grant, a trial, or a thesis and aren't sure how to group your tests, see how DASS supports researchers. We'll help you define the families and pick a correction you can defend before the data come in.
Related reading