Multiple tests: count every question behind the best result
The smallest p-value in a research exercise does not reveal how many opportunities there were to find an apparently unusual result. Count the full family of tested questions before presenting the best one. Otherwise a workflow can turn broad searching into a falsely decisive single finding.
The R p.adjust documentation describes several adjustment methods and their error-control targets. A method name is not a substitute for valid individual tests or an honestly defined family. The examples below use hypothetical p-values rather than findings about market returns.
Define the family and the error target
List every tested outcome, subgroup, horizon and comparison included in the intended claim. Preserve tests that did not look promising. A table containing only the reported winner cannot support a multiplicity correction if omitted tests helped select it.
Family-wise error rate, FWER, concerns the probability of at least one false rejection in the family. False discovery rate, FDR, concerns the expected fraction of false rejections among all rejections, conventionally assigning zero fraction when none are rejected. It is not the probability that a particular reported claim is false.
Choose the target based on the research purpose before examining results. Bonferroni can control FWER by comparing each valid p-value with alpha/m, without requiring test independence. Benjamini-Hochberg targets FDR under independence or specified positive-dependence conditions; arbitrary dependence needs appropriate treatment rather than assuming the same guarantee.
Hypothetical four-test comparison
Assume a prespecified family of four valid tests has ordered p-values 0.001, 0.020, 0.040 and 0.200. Set the nominal error target to 0.05.
| Rank | Raw p-value | Bonferroni adjusted p | BH rank threshold |
|---|---|---|---|
| 1 | 0.001 | 0.004 | 0.0125 |
| 2 | 0.020 | 0.080 | 0.0250 |
| 3 | 0.040 | 0.160 | 0.0375 |
| 4 | 0.200 | 0.800 | 0.0500 |
Bonferroni rejects only the first hypothesis at 0.05. For BH, find the largest rank k satisfying p[k] ≤ k × 0.05/4, then reject ranks through k. Here k = 2, so the first two are rejected under its applicable assumptions. Do not independently compare every row with its threshold and call that the complete step-up procedure.
The difference is not one method proving that the other made an arithmetic mistake. They address different error-control targets. Adjusted p-values also do not estimate economic effect size or the probability that a hypothesis is true.
Show the search that produced the result
For intuition, assume twenty independent tests all have true null hypotheses and each uses an unadjusted 0.05 cutoff. The chance of at least one rejection is 1 − 0.95^20, approximately 64.15%. Independence and the all-null setup are essential to this illustrative calculation; do not apply it mechanically to a correlated family.
Keep confirmatory and exploratory analyses labeled. If an unexpected subgroup suggests a useful question, report it as exploration and seek new evidence rather than disguising it as the original hypothesis. A later independent check needs a genuinely separate sample or design, not another presentation of the same selected rows.
Correcting p-values cannot repair outcome leakage, invalid standard errors, unrecorded specification searching or incorrect labels. Define the individual test correctly first. Repeatedly looking at one accumulating test also raises a stopping-rule problem that a simple count of simultaneous outcomes may not solve.
The release record should include family membership, raw p-values, adjustment method, error target, assumptions and effect estimates with relevant uncertainty. Recheck when the tested family expands or previously omitted analyses are discovered. A useful result remains transparent about all the questions asked, not only the winning one.
Check the implementation against the complete ordered vector, then restore original hypothesis identifiers after adjustment. Sorting p-values and losing their labels can attach a corrected result to the wrong claim. Include any known tests not displayed in the final chart according to the chosen procedure’s documented family-size treatment; silently correcting only selected survivors understates the search that produced them.
FAQ
Does an FDR target of 5% mean each discovery is 95% likely true?
No. It controls an expected false-discovery fraction under assumptions, not a per-claim posterior probability.
Can correction make a poorly designed test valid?
No. Multiplicity treatment relies on valid inputs and a defensible family. It does not fix biased selection or invalid uncertainty calculations.