Can I Power My Grant on My Pilot's Effect Size?

Symptoms

• Your preliminary data come from a pilot with 10 to 30 participants per group, and the effect size looked encouraging.
• The power analysis in your proposal plugs that effect size in directly, and the required sample is pleasantly small.
• A reviewer or study section asks you to justify the effect size, or notes that it seems optimistic.

What this usually means

Effect sizes from small pilots are too imprecise to anchor a power analysis. Kraemer and colleagues (2006) showed that when pilots are used this way, the most likely outcomes are that studies worth doing are abandoned and the studies that go ahead are underpowered.

The second problem has a name. Albers and Lakens (2018) call it follow-up bias: researchers go ahead only when the pilot implies a feasible sample, which happens when the pilot overestimated the effect. Even without publication bias, main studies designed this way end up with less power than planned.

A simulation in the R script below shows how large the problem is. The true effect is d = 0.40, which needs 100 participants per group for 80% power. Each pilot has 15 per group, and the main study goes ahead only if the pilot-based plan needs 150 per group or fewer:

• The middle 95% of pilot estimates ran from d = −0.33 to 1.25.
• Only 59% of pilots led to a main study. The other 41% would have abandoned a study with a real effect.
• The pilots that led to a main study estimated d = 0.67 on average, against a true 0.40.
• Those main studies had a median power of .46, and 86% had power below .80.

Common causes

1. Small samples give noisy estimates

With 15 per group, the standard error of d is around 0.37, so the estimate can easily land anywhere from no effect to a very large one.

2. Selection on promising pilots

The pilots that make a full study look affordable are disproportionately the ones that overestimated the effect.

3. Biased effect size measures

Albers and Lakens recommend against using η² from a pilot ANOVA in power analyses, because it overestimates the population effect more than ω² or ε².

4. The pilot answered a different question

Pilots often use a convenience sample, an early version of the protocol, or a short follow-up, so even an accurate estimate may not carry over to the main study.

Run these checks

1. Put a confidence interval around the pilot effect size. If it spans small to large effects, the pilot cannot anchor the power analysis.
2. Name the smallest effect worth detecting. Base it on practical or clinical importance, cost, or what would change a decision.
3. Check the literature for estimates from similar interventions and outcomes, allowing for the fact that published effects tend to be inflated.
4. Use the pilot for what it measures well: recruitment, retention, adherence, and whether the procedures work (Leon et al., 2011).
5. Check the budget ceiling: compute the power your maximum affordable sample gives across a plausible range of effect sizes.

What not to do

• Don't plug the pilot's effect size straight into a power calculator.
• Don't present the pilot's p-value as evidence that the intervention works.
• Don't use η² from the pilot.
• Don't run a pilot only to estimate the effect size.

Treatment options

Power on the smallest effect of interest

Recommended by Albers and Lakens (2018). The study is then powered for the smallest effect that would matter, whatever the pilot showed.

Safeguard power

Perugini, Gallucci, and Costantini (2014) suggest powering on the lower limit of a 60% confidence interval around the pilot estimate, which leaves an 80% chance that the true effect is at least that large. It is more conservative, but with a very small pilot it still can't fully fix the problem.

Show power across a range of effects

A short table of power at several plausible effect sizes shows reviewers that you understand the uncertainty.

Sequential or adaptive designs

Planned interim analyses with controlled error rates, or an internal pilot that becomes part of the main sample, can use the early data without wasting it (Albers & Lakens, 2018).

Explain the justification

Whatever you choose, state the effect size, where it came from, and why it is the right basis for the sample size (Lakens, 2022).

Worked example

Using the simulation above (true d = 0.40, pilots of 15 per group, a budget ceiling of 150 per group):

The safeguard approach improves power but still falls short of .80 with a pilot this small, and it rules out most studies on budget grounds. Planning on the smallest effect of interest is the only option here that guarantees the intended power for the effects that matter. The R script below reproduces the simulation.

What to tell the reviewers

Our sample size is based on the smallest effect we consider practically important, d = 0.35, rather than on the effect estimated in our pilot, because estimates from pilots of this size are imprecise and tend to overstate the effects of studies that go forward (Kraemer et al., 2006; Albers & Lakens, 2018). With 130 participants per group, the study has 80% power at α = .05 for any effect of d = 0.35 or larger. The pilot established the feasibility of recruitment, retention, and the intervention protocol, which we report in the preliminary data section.

See it in R and Python

Python uses the same exported pilot estimates and safeguard estimates as R, then independently calculates sample sizes and power using the noncentral t distribution. Download the shared pilot CSV to reproduce the original simulation summaries.

Python dependencies: NumPy and SciPy. Install with python -m pip install numpy scipy.

Download the shared synthetic CSV beside the Python script before running it. Identical seeds in R and Python do not generally generate identical samples.

# DASS Analysis Clinic, Case 003: "Can I power my grant on my pilot's effect size?"
# https://statisticalsolutions.org/analysis-clinic/pilot-effect-size-power
#
# Base R only. Simulates the common plan: run a small pilot, estimate d,
# power the main study on that estimate, and go ahead only if the required
# sample is affordable. Then compares the power the main study actually has.

set.seed(2026)
true_d   <- 0.40   # the real effect, unknown to the researcher
n_pilot  <- 15     # per group
max_n    <- 150    # largest main-study sample per group the budget allows
n_sims   <- 5000

planned_n <- function(d, power = 0.80) {
  if (d <= 0.05) return(Inf)
  ceiling(power.t.test(delta = d, sd = 1, power = power, sig.level = 0.05)$n)
}
true_power <- function(n) power.t.test(n = n, delta = true_d, sd = 1, sig.level = 0.05)$power

one_pilot <- function() {
  x <- rnorm(n_pilot, true_d); y <- rnorm(n_pilot, 0)
  sp <- sqrt((var(x) + var(y)) / 2)
  d_hat <- (mean(x) - mean(y)) / sp
  # Safeguard effect size: lower limit of a 60% CI for d (Perugini et al., 2014)
  se_d <- sqrt(2 / n_pilot + d_hat^2 / (4 * n_pilot))
  d_safe <- d_hat - qnorm(0.80) * se_d
  c(d_hat = d_hat, d_safe = d_safe)
}
pilots <- t(replicate(n_sims, one_pilot()))

n_naive <- sapply(pilots[, "d_hat"], planned_n)
go      <- n_naive <= max_n                    # follow-up only if affordable
power_naive <- sapply(n_naive[go], true_power)

cat(sprintf("True d = %.2f; n needed for 80%% power = %d per group\n", true_d, planned_n(true_d)))
cat(sprintf("Pilot d estimates: middle 95%% from %.2f to %.2f\n",
            quantile(pilots[, "d_hat"], .025), quantile(pilots[, "d_hat"], .975)))
cat(sprintf("Pilots that lead to a main study (planned n <= %d): %.0f%%\n", max_n, 100 * mean(go)))
cat(sprintf("Of those main studies: median true power %.2f; %.0f%% have power below .80\n",
            median(power_naive), 100 * mean(power_naive < 0.80)))
cat(sprintf("Promising pilots overestimate: mean pilot d among go-ahead studies = %.2f\n",
            mean(pilots[go, "d_hat"])))

# Alternative 1: power on the smallest effect size of interest (SESOI), chosen
# from what would matter in practice, e.g. d = 0.35.
cat(sprintf("Powering on a SESOI of d = 0.35: n = %d per group\n", planned_n(0.35)))

# Alternative 2: safeguard power, using the lower 60%% CI limit from the pilot.
n_safe <- sapply(pilots[, "d_safe"], planned_n)
go_safe <- n_safe <= max_n
cat(sprintf("Safeguard plan: %.0f%% of pilots fit the budget; median true power %.2f among them\n",
            100 * mean(go_safe), median(sapply(n_safe[go_safe], true_power))))

# Maintainer-only export; run from repository root. Synthetic data only.
if (Sys.getenv("CLINIC_EXPORT_DATA") == "1") {
write.csv(data.frame(d_hat=pilots[, 'd_hat'],d_safe=pilots[, 'd_safe']), 'analysis-clinic/examples/case-003-shared-data.csv',row.names=FALSE,quote=FALSE)
}

# DASS Analysis Clinic Case 003; Python dependencies: numpy, scipy.
from pathlib import Path
import numpy as np
from scipy import stats, optimize
HERE = Path(__file__).resolve().parent

def ols(X, y):
    b = np.linalg.lstsq(X, y, rcond=None)[0]
    residual = y - X @ b
    df = len(y) - X.shape[1]
    cov = (residual @ residual / df) * np.linalg.inv(X.T @ X)
    se = np.sqrt(np.diag(cov))
    ci = np.column_stack((b-stats.t.ppf(.975, df)*se,b+stats.t.ppf(.975,df)*se))
    return b, cov, ci

def power(n, d):
    # Matches R power.t.test(strict=FALSE): rejection tail in effect direction.
    return stats.nct.sf(stats.t.ppf(.975,2*n-2),2*n-2,d*np.sqrt(n/2))

pilots=np.genfromtxt(HERE/'case-003-shared-data.csv',delimiter=',',names=True)
def planned(d):
 if d<=.05:return np.inf
 return np.ceil(optimize.brentq(lambda n:power(n,d)-.8,2,1e6))
naive=np.array([planned(d) for d in pilots['d_hat']]); safe=np.array([planned(d) for d in pilots['d_safe']])
go=naive<=150; go_safe=safe<=150
powers=power(naive[go],.4)
print(f'True d=.40: {planned(.4):.0f} per group; pilot 95% range={np.quantile(pilots["d_hat"],[.025,.975])}')
print(f'Go-ahead={go.mean():.3f}; mean selected d={pilots["d_hat"][go].mean():.3f}; median power={np.median(powers):.3f}; below .80={np.mean(powers<.8):.3f}')
print(f'SESOI .35: {planned(.35):.0f} per group; safeguard go-ahead={go_safe.mean():.3f}; median power={np.median(power(safe[go_safe],.4)):.3f}')
assert planned(.4)==100 and planned(.35)==130

Download Python script · Download shared synthetic CSV

Sources

• Albers, C., & Lakens, D. (2018). When power analyses based on pilot data are biased: Inaccurate effect size estimators and follow-up bias. Journal of Experimental Social Psychology, 74, 187–195.
• Kraemer, H. C., Mintz, J., Noda, A., Tinklenberg, J., & Yesavage, J. A. (2006). Caution regarding the use of pilot studies to guide power calculations for study proposals. Archives of General Psychiatry, 63(5), 484–489.
• Lakens, D. (2022). Sample size justification. Collabra: Psychology, 8(1), 33267.
• Leon, A. C., Davis, L. L., & Kraemer, H. C. (2011). The role and interpretation of pilot studies in clinical research. Journal of Psychiatric Research, 45(5), 626–629.
• Perugini, M., Gallucci, M., & Costantini, G. (2014). Safeguard power as a protection against imprecise power estimates. Perspectives on Psychological Science, 9(3), 319–332.