Small proportions: show interval uncertainty beside the rate
A small observed error rate can remain highly uncertain when the audit sample is small. Reporting one failure among twenty checked items as a precise five-percent population error rate hides the sampling uncertainty. Show an interval and the sampling design beside the fraction.
The NIST proportion confidence-limit reference presents the Wilson interval construction. This worksheet applies that construction to a hypothetical binary document audit. It does not turn a handpicked reference set into a random sample of all future tasks.
Define the trial and target rate
Record the eligible population, sampling method, pass/fail rule and independent unit. If several fields come from one document, their errors may be correlated. Treating every field as an independent trial can exaggerate effective sample size.
For a binomial model, assume n independent trials with a common failure probability p and a fixed evaluation rule. The observed proportion is p-hat = x/n. A standard normal approximation centered on p-hat can perform poorly near zero or one, especially for small n.
For a two-sided Wilson interval, choose normal critical value z for the desired confidence level. Define D = 1 + z²/n, center = (p-hat + z²/(2n))/D, and half-width = z × sqrt(p-hat(1 − p-hat)/n + z²/(4n²))/D. The endpoints are center minus and plus half-width.
Hypothetical one-failure audit
Assume twenty fictional documents were randomly selected from a defined, compatible population and independently graded. One fails. Using z = 1.96 for a nominal 95% two-sided interval, p-hat is 0.05, D is 1.19208, the Wilson center is about 0.12251 and the half-width about 0.11363.
| Audit result | Hypothetical estimate |
|---|---|
| Observed failures | 1 of 20 |
| Point proportion | 5% |
| Wilson lower endpoint | About 0.89% |
| Wilson upper endpoint | About 23.61% |
The interval is wide and asymmetric around the observed five percent. That is not a defect to smooth away. It reflects limited information about the common failure probability under the model. More displayed decimal places do not make the sample more informative. Report the rounding applied to endpoints.
Now assume zero failures among twenty. Zero observed does not establish a zero population rate. The two-sided 95% Wilson upper endpoint is z²/(n + z²), about 16.11%. If the question instead requires an exact one-sided 95% binomial upper bound after zero failures, the expression is 1 − 0.05^(1/20), about 13.91%. These are different methods and sidedness, not contradictory answers.
Keep inference tied to selection
An interval addresses sampling variation under assumptions, not systematic selection or grading errors. A collection chosen because its documents are easy cannot establish the rate for complex documents. If the population is small and sampling is without replacement, consider whether the binomial approximation is appropriate for that design.
Preserve unresolved grades separately. Calling every ambiguous answer a pass changes both the measured outcome and its interpretation. A change in the failure definition requires rerating or separate reporting, not just recalculating a formula with the same x.
State whether the interval was specified before examining repeated audit results. Repeatedly checking until a favorable upper bound appears raises a separate stopping-rule problem. A single fixed-sample interval does not automatically remain valid under arbitrary repeated selection.
The final audit record should include x, n, sampling frame, interval method, confidence level, sidedness and excluded cases. Recheck after the workflow, population or grading rule changes. It is a bounded estimate of performance on the stated population, not certification of every future answer.
A release threshold should be specified before the audit. If the requirement is an upper confidence bound below 10%, the observed five-percent point estimate in the example does not satisfy it under the stated Wilson interval. Do not replace the rule with a point-estimate threshold after seeing the wide interval. Save the decision rule alongside the numeric audit so future reviewers can distinguish a changed standard from improved evidence.
FAQ
Do zero failures prove the workflow has no errors?
No. A finite sample can observe zero even when the population failure probability is positive.
Can I put a confidence interval on a convenient test set?
You can compute a number, but population inference needs a defensible sampling model. Label diagnostic cases without implying random representativeness.