Aggregate reversals: test within-group evidence before a claim
An aggregate success rate can deteriorate even when every measured subgroup improves. The change may reflect a shift toward harder cases rather than worse within-group performance. Research should reconstruct the subgroup denominators before asking AI to explain the headline reversal.
The Stanford mathematical analysis of Simpson’s paradox discusses how aggregate and subgroup associations can differ. The fictional research-workflow example below supplies its own counts and decomposition. It does not establish a causal effect of any real model or prompt change.
Preserve counts as well as rates
Choose strata that correspond to the research question before selecting a convenient reversal. Record successes and eligible cases for each period and stratum. Verify that task definitions, grading rules and coverage are comparable. If “complex document” means something different in each period, the apparent within-group comparison may also be misleading.
Express the aggregate rate as the sum of stratum rate multiplied by its population share. The same formula yields different aggregates when shares change, even if every stratum rate is stable. Save the actual numerator and denominator instead of retaining only rounded percentages.
Ask separately about performance on the observed workload and performance on a fixed workload mix. Both can be useful, but they are different estimands. A standardized comparison should name its reference composition rather than claiming to be the uniquely correct overall number.
Hypothetical easy and complex cases
Assume a fictional extraction workflow processes easy and complex documents. In period A it passes 90 of 100 easy cases and 2 of 10 complex cases. In period B it passes 19 of 20 easy cases and 30 of 100 complex cases.
| Stratum | Period A | Period B |
|---|---|---|
| Easy | 90/100 = 90% | 19/20 = 95% |
| Complex | 2/10 = 20% | 30/100 = 30% |
| All cases | 92/110 ≈ 83.64% | 49/120 ≈ 40.83% |
Each stratum improves, yet the aggregate falls dramatically because the workload shifts toward complex cases. Saying the workflow became worse on every document type would contradict the table. Saying the observed workload became easier would also be wrong.
Hold period A’s composition fixed: easy cases have weight 100/110 and complex cases 10/110. Applying period B’s rates produces (100/110) × 95% + (10/110) × 30% ≈ 89.09%. Under this convention, within-stratum improvement raises the standardized rate by about 5.45 percentage points from period A.
The difference between period B’s observed 40.83% and the standardized 89.09% is a composition contribution under this ordering. Another decomposition can allocate interactions differently. Preserve the convention instead of presenting one attribution as a physical accounting truth.
Decide which conclusion the evidence supports
For staffing and review workload, the observed period B rate matters: more difficult incoming documents can create more human review even if the system improves. For a controlled workflow comparison, a stable task mix or matched reference set may be more informative. Do not choose the target after seeing which one is flattering.
Inspect whether strata are sufficiently homogeneous. A broad complex category can itself shift toward harder cases. More stratification is not automatically better, because small denominators increase uncertainty and post hoc categories can invite selective reporting. Report counts and retain unresolved category changes.
Standardization is descriptive unless additional design supports causal interpretation. Changes in documents, reviewers, tool versions and selection can remain within strata. A reversal does not identify which variable should be controlled or prove that grouping removes confounding.
The deliverable is the observed aggregate, subgroup evidence and a labeled fixed-composition comparison. Recheck when workload definitions or grading policies change. Keep an aggregate deterioration visible even when the standardized comparison improves, since it still describes the actual operational workload.
A standardized percentage can also change simply because the analyst chooses another reference mix. Publish the fixed weights so a reviewer can reproduce the result and distinguish that choice from the observed denominators. If a stratum is absent in one period, its rate is not observed there; inventing it to complete standardization would introduce another model assumption rather than resolve the missing comparison.
FAQ
Which rate should be reported?
Report the observed rate and, when relevant, the defined standardized comparison. Explain which decision each answers.
Does the reversal prove a model improvement caused better results?
No. The arithmetic separates composition from within-stratum rates; causal attribution requires further design and evidence.