Equivalence checks: define acceptable differences before testing
Failing to detect a difference does not establish that two research workflows are close enough for the intended use. A noisy comparison can fail to reject zero while still allowing differences that matter. An equivalence study reverses the evidentiary task: define the acceptable mean difference first, then ask whether the data exclude differences at or beyond those bounds.
The deliverable is a prespecified margin, a correctly paired analysis and a decision table that separates mean equivalence, evidence of a nonzero difference, and insufficient precision. Mean equivalence does not establish that individual document results are interchangeable.
Define the difference and the margin before collection
Suppose an established workflow and a new workflow process the same archived documents. The measured outcome is completion time in minutes. Define d as new minus old, so positive values mean the new workflow is slower. For this hypothetical operational decision, stakeholders decide in advance that an average change strictly between −2 and +2 minutes is acceptable.
The two-minute margin needs a practical explanation: the downstream schedule can tolerate that average change while preserving the required work. It must not be selected because a trial interval happened to fit inside it. Record the eligible document population, timing rules, handling of failed runs and the exact estimand. If the actual requirement is that nearly every individual document stays within two minutes, this mean-equivalence design answers the wrong question.
The official statsmodels paired TOST documentation identifies a two-one-sided-t-test procedure for a paired mean difference. Its difference direction is x1−x2; use x1=new and x2=old for the convention above. Keep the paired identity even if a parameter description elsewhere on the page uses generic independent-sample wording.
Write the hypotheses explicitly
For population mean difference mu, the nonequivalence null is mu ≤ −2 OR mu ≥ 2. The equivalence alternative is −2 < mu < 2. At alpha 0.05, both one-sided tests must reject their respective boundary nulls.
For n independent document pairs and an approximately normal difference model, let SE = SD(d)/sqrt(n), with n −1 degrees of freedom. The test statistics are:
t_lower = [mean(d)+2]/SE, tested in the upper tail;
t_upper = [mean(d)−2]/SE, tested in the lower tail.
The combined TOST p value is the larger of the two one-sided p values. It is not their sum, and passing just one boundary is insufficient. The equivalent interval rule is that the two-sided 90% t confidence interval lies strictly inside [−2,2]. Each one-sided test remains at 5%; this is not a request to split alpha again into 2.5% for these two tests.
Compare two studies that both fail a zero-difference test
Use n = 25 pairs, hence 24 degrees of freedom. The 95th percentile of that t distribution is about 1.710882. Construct two sample summaries with the following standard errors:
| Study | Mean difference | SE | 90% interval | Mean-equivalence result |
|---|---|---|---|---|
| A | 0 | 0.584494 | [−1,1] | inside both bounds |
| B | −1.5 | 1.461235 | [−4,1] | lower bound not excluded |
For A, the 90% half-width is 1.710882×0.584494=1. Both one-sided statistics are equally far beyond their rejection boundaries, so A supports the prespecified average equivalence.
For B, the 90% half-width is 2.5. Its interval includes changes below −2, so it cannot establish equivalence even though it also allows zero. The lower-bound test is the limiting one: (−1.5+2)/1.461235≈0.342, which does not reject at 5%. Failure to establish equivalence is not proof that the actual mean lies outside the bounds.
Using the 24-degree-of-freedom 97.5th percentile, about 2.063899, the ordinary 95% mean-difference intervals are approximately [−1.206,1.206] for A and [−4.516,1.516] for B. Both include zero. An ordinary “no statistically significant difference” label would collapse these very different levels of precision into the same statement.
The constructed SEs correspond to paired-difference SDs equal to five times the SE, since sqrt(25)=5. They are not independent-group standard errors. Preserve the raw paired differences in a real study; a summary table alone cannot diagnose skew, dependence or selective failures.
A detectable difference can still be acceptably small
For a third study C, keep n = 25 and the same margin but set mean difference 0.5 and SE 0.1. Its 90% interval is approximately [0.329,0.671], entirely within [−2,2]. Its ordinary 95% interval is [0.294,0.706], excluding zero.
Thus C can show both a nonzero average difference and equivalence under the chosen practical margin. Those conclusions do not conflict. One concerns exactly zero; the other concerns differences large enough to be operationally unacceptable. Report “a small positive average time difference, within the prespecified mean margin” rather than forcing a binary same/different label.
These examples do not validate the two-minute margin. A narrower practical requirement could change the equivalence decision without changing any observations. That is why margin justification belongs in the design record and should not be reverse-engineered from a p value.
Keep individual disagreement and missingness visible
Averaging can hide cancellation. Some documents might take much longer with the new workflow and others much less time, yielding a small mean. That can support mean equivalence while making substitution unsuitable for documents facing strict individual deadlines. Inspect the difference distribution and use an agreement analysis with a separately justified individual tolerance if that is the actual decision.
Naturally paired data also require independent pairs at the intended analysis level. Twenty-five documents drawn repeatedly from one shared task may not supply the independence assumed by the simple t calculation. Timing both workflows in a fixed order can introduce practice or caching effects. Plan order and execution conditions rather than treating workflow names as the only difference.
Failed or missing runs need a rule that is fixed before collection. Dropping only failures of the new method can make completion times look equivalent among a convenient subset. Report the analyzed population and failure counts alongside time results. A complete-pair analysis does not establish equivalence of success rates or accuracy.
Execute a reviewable equivalence check
| Stage | Required evidence | If the requirement fails |
|---|---|---|
| Specify | target mean, direction, units and margins | pause the equivalence claim |
| Collect | matched IDs, versions, timing and failure status | repair joins without concealing failures |
| Diagnose | difference distribution and dependence structure | revise the model or design |
| Calculate | mean, SD, SE, df, both one-sided tests | retain both boundary results |
| Cross-check | 90% interval versus original margins | investigate disagreement with the tests |
| Report | equivalence decision plus 95% difference interval | keep precision and practical meaning distinct |
When software outputs disagree, check pair order, weighting, transformation, missing-value treatment and the sign of the difference before changing the margin. The function documentation is a method reference; the calculations here are independently reproducible t-distribution examples, not a claim that a particular package was executed on real workflow data.
New independent pairs collected under a prespecified plan may improve precision and resolve an inconclusive comparison. A changed workflow version, timing definition or acceptable margin instead creates a revised study specification. Preserve the original decision and date the update. Repeatedly collecting until an interval happens to fit requires its own sequential design; elapsed data collection alone does not justify treating repeated tests as a single fixed study.
Check a boundary case without rounding it into a pass
Construct another n = 25 summary with mean −0.5 and SE=1.5/1.710882≈0.876741. Its 90% interval is [−2,1]. The lower one-sided test sits at the 5% boundary: its statistic is the 95th percentile of t24. With the strict interior convention used here, touching the margin does not establish equivalence. A displayed endpoint of −2.00 can also conceal a value just inside or outside, so evaluate unrounded calculations rather than the formatted interval.
The boundary example is not a fourth practical margin. It tests whether the implementation respects the decision rule already specified. Preserve numeric precision, the comparison convention and both tail probabilities in the audit output. If software and an interval check disagree, that discrepancy should trigger a reproducibility check, not a retrospective choice of whichever representation reports equivalence. The same discipline applies to study A: it supports the stated average target, not an unspecified stronger claim about every document.
The margin record should identify its owner and intended decision as well as its number. If reviewers disagree about acceptable average delay, resolve that operational disagreement before treating a statistical test as decisive. The same data can legitimately support one prespecified use and remain inadequate for another; preserving the specifications makes the difference explainable rather than arbitrary.
FAQ
Does a large ordinary difference-test p value prove equivalence?
No. Study B cannot reject zero yet still allows unacceptable mean differences. Equivalence requires excluding both prespecified boundaries.
Does passing TOST mean every document is interchangeable?
No. The stated TOST concerns the population mean difference. Individual disagreement and failure rates require separate evidence.