Measurement agreement: high correlation can hide unusable gaps
Two extraction methods can rank documents almost identically and still disagree too much to replace one another. Correlation addresses co-movement. Interchangeability asks whether the difference for an individual document is small enough for the intended use. A test of average difference asks a third question and cannot settle the second one by itself.
The research output is a paired-difference worksheet with a prespecified tolerance, estimated bias, spread of individual differences and explicit model checks. Neither method is automatically the truth. A comparison can establish how they differ while leaving their common error against an independent reference unresolved.
Start with the replacement decision
In this hypothetical comparison, two workflows measure the same numerical field, in the same units, from archived documents. Define A as the established workflow and B as the candidate. Set d = B − A, so a positive difference means B reports a larger value. Before looking at results, suppose the use case permits individual differences only within −5 to +5 units. The five-unit bound is an operational assumption for this example, not a universal statistical threshold.
Collect both readings on the same document version. Keep document identifiers, raw A/B pairs, units, order and method versions. A missing B value is not a zero difference. If failures occur mostly in difficult documents, restricting the analysis to easy complete pairs changes the population and must be reported.
In Bland and Altman’s original agreement paper, consult the discussion of correlation and the section on plotting differences against pair means. Its clinical examples motivate the method; the document measurements and numerical cases below are original illustrations, not clinical evidence or a certification standard.
A perfect correlation that fails immediately
Take six A readings 10,20,30,40,50,60. Let B be 20,30,40,50,60,70. The correlation is exactly 1 because B=A+10. Yet every difference is +10, twice the allowed absolute gap. Perfect rank and linear agreement have not produced usable numerical agreement.
The wide range of underlying levels helps produce impressive correlation while saying nothing about whether ten units is tolerable. Expanding that range would not fix any document. In this constructed case the difference SD is zero, but the fixed bias alone makes replacement unacceptable under the stated criterion. Do not generalize that constant gap to actual data without checking it.
Calculate bias and limits of agreement
For a separate six-document sample, suppose the observed differences are:
−2, 0, 2, 2, 4, 6.
Their sum is 12 and mean difference is 2. Deviations from that mean are −4,−2,0,0,2,4, whose squared sum is 40. Therefore the sample difference variance is 40/5=8 and the difference SD is √8=2.828. This is the spread of document-level differences, not the standard error of the mean difference.
Under an approximately normal, constant-variance difference model, the usual plug-in limits are:
mean(d) ± 1.96 × SD(d) = 2 ± 5.544 = [−3.544, 7.544].
These estimated limits extend above the prespecified upper tolerance +5. The sample does not support the claim that typical individual disagreement is confined to the required band. One observed difference, 6, also exceeds the band, but saying “five of six passed” is not equivalent to establishing 95% population coverage.
Six pairs are deliberately small so the arithmetic can be checked. They are not enough for a confident production acceptance decision. The limits themselves have sampling uncertainty. An acceptance plan should state how uncertainty around the limits is assessed and how close a limit may lie to a tolerance before more evidence is required. Drawing two lines and calling them guaranteed boundaries is inadequate.
The numerical difference list does not establish constant variance or normality. Retain the original pair means and collection order to evaluate those assumptions; they cannot be recovered from six differences alone. If only the difference summary is available, the calculation remains a provisional demonstration.
Do not substitute the confidence interval for mean bias
For the same six differences, the standard error of their mean is sqrt(8)/sqrt(6)=1.155. With five degrees of freedom, a two-sided 95% t interval uses a critical value about 2.571. It gives:
2 ± 2.571 × 1.155 ≈ [−0.968, 4.968].
That interval contains zero. It does not show the methods are interchangeable. It is uncertainty about the population mean difference, while the agreement limits describe the modeled spread of individual differences. Substituting SE for SD in the agreement calculation would make disagreement look artificially small.
It is also possible to estimate a small nonzero bias very precisely with many pairs while individual differences remain too wide. Whether the average bias can be corrected is separate from whether residual disagreement after correction meets the use case. An offset correction needs its own validation; subtracting the sample mean cannot retrospectively make the original methods agree.
Change the model when the error mechanism changes
An independent boundary case is B=1.1A for positive measurements. At A 10 the difference is 1; at A 100 it is 10. A single constant-unit spread may conceal that disagreement grows with level. Plotting difference against pair mean helps reveal this structure; correlation can still equal 1.
For strictly positive values, a prespecified ratio target may justify examining log differences. Here log(B)−log(A)=log(1.1)≈0.09531, corresponding to a ratio 1.1. A ratio tolerance must be justified in its own units before analysis. Taking logs after seeing failure and inventing a wider target does not validate replacement. Zeros and negative values also prevent this simple log-ratio treatment.
Repeated measurements create another boundary. Ten readings on one document are not ten independent documents. Separate repeatability from between-document agreement, define whether the proposed replacement uses a single reading or an average, and use an analysis consistent with that collection structure. Otherwise, a large row count can hide a small effective comparison study.
Finally, both methods could share the same source error and agree extremely well while being wrong. Agreement against another workflow is not accuracy against truth. An independent reference sample is needed for that additional claim.
Run the agreement review in a fixed order
| Step | Evidence or calculation | Required response to a problem |
|---|---|---|
| Fix the intended use | field, units, population and tolerance | resolve the target before testing |
| Match observations | identical document versions and pair IDs | keep failures and unmatched records visible |
| Inspect differences | d versus pair mean and collection order | investigate level effects and regime changes |
| Estimate quantities | mean, difference SD and agreement limits | preserve formulas and degrees of freedom |
| Assess uncertainty | precision of limits and model adequacy | classify insufficient evidence explicitly |
| Decide on replacement | limits versus the original tolerance | distinguish failure from an unresolved study |
If two exports disagree, reproduce the sign convention and raw pairing first. A reversed sign flips the bias and both limits, while a mistaken join can manufacture a spread unrelated to method disagreement. Do not average incompatible units or silently discard large gaps as outliers without checking the source document.
Update the decision when new independently sampled pairs improve precision, a unit or join error is corrected, or a method version changes. A changed tolerance is a changed decision specification and should be dated separately. The final report should state what remains untested: agreement at unrepresented levels, difficult missing documents, or accuracy against an external reference. This makes a replacement decision reviewable without overstating what a high correlation or a nonsignificant mean test can establish.
Preserve the evidence behind a replacement status
Use at least three statuses in the decision file: supported within the prespecified model and tolerance, observed evidence against the requirement, and unresolved because precision or coverage is insufficient. A small study with apparently narrow plug-in limits should not receive the first status merely because no observed pair crossed the tolerance. The limits may be estimated imprecisely, and the sample may omit the most difficult documents.
Attach every large difference to its source excerpt and unit conversion. If a transcription error is corrected, retain both the original and corrected pair with the reason; if the measurement was valid, do not discard it to improve agreement. A preplanned subgroup can identify a range where replacement works, but a subgroup selected after inspecting failures needs fresh validation. This preserves a useful restricted-use conclusion without quietly turning a failed all-document claim into a successful one about a handpicked subset.
Keep the planned direction of substitution in the report. A candidate method checked on routine documents should not inherit approval for unusual formats merely because its average bias resembles that of the routine group. Save a coverage list of represented formats and levels so the next sampling round can target the missing conditions.
FAQ
Does a mean-difference interval containing zero establish agreement?
No. It concerns average bias. Individual differences can still exceed the tolerance, as the six-pair calculation illustrates.
Are limits of agreement hard bounds on every future case?
No. They are model-based estimated limits, with sampling uncertainty. They neither cover every individual nor establish a guaranteed population proportion.