Paired comparisons: preserve the match before measuring change
When the same objects are measured twice, uncertainty about change depends on the differences within those matched objects. Treating the two columns as unrelated samples discards the pairing and can misstate precision. Preserve the observation key before comparing a revised research workflow with its earlier version.
The NIST paired-observation reference defines natural matches and calculates statistics of their differences. Natural pairing is a feature of the design, not something created by sorting two unrelated columns until their values look similar.
Define the pair and direction
Record a stable object identifier, measurement conditions and before/after values. State whether d means after minus before or the reverse. A positive difference can represent improvement for one metric and deterioration for another; the sign needs an economic interpretation.
For n complete pairs, compute each d, the mean difference d-bar and the sample standard deviation s[d]. Under independent pairs with suitable assumptions, the estimated standard error of the mean difference is s[d]/sqrt(n). The independence requirement concerns pairs; measurements inside each pair are deliberately related.
The variance identity is Var(after − before) = Var(after) + Var(before) − 2Cov(after,before). Ignoring covariance can lose precision when it is positive, but can understate uncertainty if covariance is negative. Do not assume pairing always makes intervals narrower regardless of the actual relationship.
Hypothetical four-document comparison
Assume a fictional researcher times the same four documents under two workflows. Minutes are measured consistently, and larger values mean slower completion.
| Document | Before | After | After minus before |
|---|---|---|---|
| A | 10 | 11 | +1 |
| B | 20 | 22 | +2 |
| C | 30 | 29 | −1 |
| D | 40 | 42 | +2 |
The original means are 25 and 26 minutes. Differences are 1, 2, minus 1 and 2, giving mean change +1 minute. Squared deviations from that mean sum to 6, so sample difference variance is 6/3 = 2 and standard deviation is sqrt(2), about 1.4142 minutes. The paired standard error is about 0.7071 minutes.
An independent-samples calculation would use the separate column variances. Those are approximately 166.6667 and 168.6667, producing sqrt(166.6667/4 + 168.6667/4), about 9.1561 minutes. The difference arises because document difficulty contributes to both columns and cancels substantially within each pair.
These invented four pairs illustrate arithmetic, not a reliable estimate of any workflow’s general speed. Small-sample intervals need the relevant assumptions, and four documents cannot demonstrate performance across all formats or tasks.
Keep the design intact
Check missing pairs before calculation. An object present only before does not have an observed change. Do not fill its after value with the group’s average merely to preserve n. Report complete pairs, unmatched cases and whether missingness could select easier or harder objects.
Preserve order and carryover effects. If the same reviewer always uses the old workflow first, familiarity can affect the second timing. Matching documents controls their identity, not practice effects, changed reviewers or altered instructions. A clock-resolution change also requires measurement review. A stronger experimental design might address those issues separately.
For comparisons across companies or instruments, a shared row number is not a natural match. Matching on characteristics creates a design requiring justification and different causal assumptions. This worksheet’s arithmetic does not establish that a matched control is a valid counterfactual.
Release the pair key, sign convention, difference summary and design limits. Recheck after deduplication, identifier repairs or changes to the measured task. A correctly paired descriptive change still does not prove what caused it.
A useful validation check is that the mean of complete-pair differences equals the difference of the two complete-pair means. If it does not, examine row alignment, missing-value filters and weights before interpreting the result. This identity applies to the same equally weighted pairs, not to two columns filtered differently. Weighted paired analysis also needs a stated estimator and appropriate variance treatment, rather than adding weights to one side only.
FAQ
Can two columns be paired because they have equal row counts?
No. Each row must identify a meaningful match established by the design or an explicitly justified matching rule.
Does paired analysis prove a workflow caused improvement?
No. It measures within-pair change. Order, time and other changing conditions can still explain the result.