Study power: choose the difference worth detecting in advance
A study budget should be connected to a difference worth detecting before outcomes are inspected. “We have many rows” is not a design specification. The relevant inputs include the independent sampling unit, expected error scale, false-positive threshold, target power and smallest substantively important difference.
This worksheet plans a comparison of two document-research workflows. It does not optimize an investment strategy or select a trading rule. Its output is a prospective sample-size and minimum-detectable-difference table, with assumptions and recruitment losses visible.
Define the study before solving for n
Suppose independent documents are randomly allocated to workflow A or B, and the outcome is completion time in minutes under a fixed timing rule. The primary estimand is the population mean time difference. Each group has the same planned number n of independent documents, with common within-group SD 10 minutes. Use a two-sided test, alpha 0.05 and target power 0.80 at a specified nonzero difference.
Power is the probability of rejection under the assumed alternative and design. It is not the probability that a hypothesis is true, nor a guarantee that a particular completed study will be significant. At the planned difference, 80% power still leaves 20% nonrejection under those assumptions.
The official TTestIndPower.solve_power documentation specifies a two-sample t-test calculation with standardized effect size, first-group observations, alpha, power and allocation ratio. Here ratio=1 and the standardized effect is the mean difference divided by 10. The function’s nobs1 is the first group’s size, not the combined total. Save these inputs rather than only a solver’s final number.
Compute a transparent normal approximation
For two equally sized independent groups with common SD sigma, the mean-difference standard error is sigma × sqrt(2/n). Using rounded normal critical values 1.96 and 0.84 gives the common planning approximation:
n ≈ 2 × (1.96+0.84)² × sigma² / delta².
With sigma 10, the numerator is 2×2.8²×100=1568. Therefore:
| Difference delta | Standardized difference | Approximate n per group | Rounded up | Total analyzed |
|---|---|---|---|---|
| 5 minutes | 0.5 | 62.72 | 63 | 126 |
| 2 minutes | 0.2 | 392.00 | 392 | 784 |
The smaller target is not merely a little more demanding. Holding everything else fixed, required size scales approximately with the inverse squared difference. Detecting 2 rather than 5 minutes needs 6.25 times the approximate sample size: (5/2)². This comparison is useful when deciding whether the research budget addresses the operational question at all.
The expression uses rounded quantiles and a large-sample approximation. Do not describe 63 as an exact t-test requirement, and do not imply that every group with that many observations has precisely 80% power.
Check the t design and distinguish two uncertainties
When the common SD is estimated under an independent normal, equal-variance model, use the noncentral t distribution for a more specific calculation. With n per group, degrees of freedom are 2n −2 and the noncentrality parameter is d×sqrt(n/2), where d is the standardized difference. Evaluate both rejection tails beyond the central-t critical values.
Independently reproducing that calculation gives continuous sample-size solutions about 63.766 for d 0.5 and 393.406 for d 0.2. Rounding upward yields 64 and 394 per group, respectively, or 128 and 788 total analyzed documents. These model-based planning numbers use the stated t distribution rather than the rounded normal approximation.
The t calculation accounts for estimating the SD in that model. It does not remove uncertainty in the planning assumption that the population SD is 10. That input needs a relevant pilot, external evidence or a labeled range. A pilot with an optimistic SD can understate the required main-study size even when the solver is numerically correct.
For example, in the rounded normal approximation with delta 5, using SD 8 gives n = 40.1408, rounded 41 per group. Using SD 12 gives n = 90.3168, rounded 91. This is an original design-sensitivity table, not a probability interval for the true SD. Keep the evidence supporting each scale scenario alongside the calculation.
Turn a fixed budget into a detectable difference
If the analyzed budget is fixed, invert the same rounded normal expression:
MDE ≈ 2.8 × 10 × sqrt(2/n).
| Available n per group | Approximate MDE in minutes |
|---|---|
| 50 | 5.600 |
| 100 | 3.960 |
| 400 | 1.980 |
Doubling 50 to 100 does not halve the detectable difference; it reduces it by the square-root factor. Roughly quadrupling the independent sample halves the MDE. These are characteristics of the stated design, not reasons to redefine an important two-minute difference as unimportant because the available study cannot detect it.
If the practical target is 2 minutes but the budget permits only 50 per group, the honest plan is underpowered for that target under the assumptions. Options include changing the collection design, improving measurement precision or increasing budget. Do not substitute a larger target after looking at favorable sample results and call it the original study question.
Recruit for usable observations without assuming missingness away
Suppose the chosen t design requires 64 analyzed documents per group and 20% are expected to be unusable. A simple recruitment allowance is ceil(64/0.8)=80 per group. For the 394-per-group design, it is ceil(394/0.8)=493 per group. These allowances plan expected usable counts; random losses can still leave fewer than required.
Inflating recruitment does not fix selective missingness. If B fails mainly on long documents, comparing only completed cases changes the estimand and can bias the difference. Define failures, timeouts and unavailable source documents before allocation. Report their counts and causes as well as the analyzed timing sample.
Independence also matters more than row count. Repeated runs of one document are not new independent documents. Several documents from one tightly shared task may require a clustered design. If each document naturally receives both workflows, a paired design uses the variance of within-document differences and requires a different power calculation. The two-independent-group formula should not be retained merely because it is convenient.
Keep prospective planning separate from completed-study evidence
After collection, report the estimated difference, its interval, failures and deviations from the design. Recomputing “observed power” using the observed effect as though it were the planned true effect does not supply independent evidence for the result. A nonsignificant outcome can be inconclusive; a large planned power does not establish equivalence or prove no meaningful difference.
Likewise, power does not repair confounding, incorrect joins, changing measurement definitions or an unrepresentative sample. Specify one primary comparison or an appropriate multiplicity plan before treating alpha 0.05 as the study’s error control. Trying many outcomes and reporting the one that meets its target is a different procedure.
| Planning item | Evidence to preserve | Revision trigger |
|---|---|---|
| Practical difference | decision rationale in minutes | intended use changes |
| Error scale | pilot provenance and SD scenarios | new comparable pilot evidence |
| Sampling unit | document/task hierarchy and allocation | dependence or pairing discovered |
| Test specification | alpha, tails, variance model and power | design changes before outcomes |
| Recruitment | analyzed target and loss allowance | updated loss evidence |
| Final reporting | estimate, interval and protocol deviations | data or measurement errors corrected |
A revised plan should retain its original date and explain whether outcomes were already available. This lets reviewers distinguish genuine prospective design from post-result justification. The calculation’s value is to expose a mismatch between question and evidence early enough to change the study, while preserving a clear record of what the completed data can actually answer.
Check allocation before counting the total budget
With equal within-group SD 10 and a fixed total 120 independent documents, a 60/60 allocation gives mean-difference variance 100/60+100/60≈3.333. A 40/80 allocation gives 100/40+100/80=3.75. Their approximate standard errors are 1.826 and 1.936, respectively. Using the same rounded 2.8 planning factor gives MDEs about 5.112 and 5.422 minutes. Equal total size does not imply equal information about a difference.
The larger group cannot entirely compensate for a very small comparison group. Unequal costs, recruitment limits or a different variance structure can justify an unequal allocation, but those reasons belong in the design and the solver’s ratio input. Do not pool both groups into a single 120-observation sample-size field. The example also reinforces the distinction between total recruitment, usable observations and group-specific allocation: each answers a different budgeting question, and a solver needs the intended analyzed design.
Retain the calculation version when a pilot updates the assumed SD. A reviewer should be able to reproduce both the original budget and the revised one, including rounding and allocation. If recruitment has already begun, document the permitted update rule before using new information to change the final sample target.
FAQ
Does 80% power mean an 80% chance the hypothesis is correct?
No. It is a rejection probability under a specified alternative, sampling model and test design.
Can repeated measurements be counted as additional independent documents?
No. They can improve measurement precision under an appropriate model, but their dependence must remain in the study design.