Repeated measurements: separate within-run and between-day error
Repeating a measurement twelve times on one afternoon does not provide the same evidence as measuring four times on each of three days. The second design can expose changes shared by an entire day’s results. The first design cannot separate that component from the object’s level, however many rows appear in the spreadsheet.
For a research measurement process, the deliverable is a variance budget tied to a collection design: within-day repeatability, variation between day means, and the estimated day component after removing the within-day contribution. This is a continuous-measurement problem, distinct from categorical agreement between annotators.
Freeze the object and define a repetition
Imagine repeatedly reading a numerical level from one archived chart using a fixed extraction workflow. Preserve the exact chart bytes, scale and intended quantity. The underlying object must remain unchanged. Re-reading an issuer’s newly updated chart each day would mix genuine information changes with measurement variation.
Define a repetition as a fresh measurement operation. Reopening a cached result or copying the same output into four rows supplies no new repeatability evidence. Record the day, repetition, software version, operator and settings. If a model-based extractor is used, record its relevant execution conditions as well. Decide whether the intended process includes different operators; do not silently add that source after seeing a troublesome result.
NIST’s repeatability analysis, particularly its pooling rules, combines within-group variances using their degrees of freedom. Its measurement setting is a gauge study; the archived-chart worksheet below is an original illustration of the same nested-design accounting, not NIST certification of a financial-data workflow.
Work through a three-day, four-repetition example
Assume an additive random-effects model:
y[d,j] = theta + D[d] + e[d,j].
Here theta is the fixed object’s level, D is a zero-mean day effect with variance tau², and e is a zero-mean within-day error with variance sigma². Day effects are independent across days; within-day errors are mutually independent and independent of the day effects. Variances are stable across the three days. Those are modeling assumptions to evaluate, not properties guaranteed by the labels in the dataset.
Construct four readings per day by adding deviations −sqrt(3), −sqrt(3), +sqrt(3), +sqrt(3) to the following day means:
| Day | Mean of four readings | Within-day sum of squares | Within-day degrees of freedom |
|---|---|---|---|
| 1 | 47 | 12 | 3 |
| 2 | 50 | 12 | 3 |
| 3 | 53 | 12 | 3 |
Each within-day variance is 12/3=4. Pool the sums of squares, not the standard deviations: sigma_hat² = (12+12+12)/(3+3+3) = 4, giving within-day SD 2 and nine within-day degrees of freedom. Keeping the construction in square-root form avoids rounding the raw readings before calculating the variance.
The grand mean is 50. The sample variance of the three day means is (9+0+9)/(3−1)=9, giving day-mean SD 3 with only two degrees of freedom. That 9 includes the day effect and the residual variance of a four-reading mean. Since the latter is 4/4=1, the estimated day component is:
tau_hat² = 9 − 4/4 = 8.
NIST’s reproducibility page, at “Relationship to day effect,” gives this subtraction structure. Do not label the observed day-mean variance 9 as the isolated day-effect variance 8; the quantities answer different questions.
Let the reporting target determine the variance
Under the stated model and using those plug-in components, compare three outputs:
| Reported result | Variance | Standard deviation |
|---|---|---|
| One reading on a randomly selected day | 8+4=12 | 3.464 |
| Mean of four readings on one day | 8+4/4=9 | 3.000 |
| Mean across three days, four readings each | 8/3+4/12=3 | 1.732 |
The last row averages three independent day effects as well as twelve within-day errors. It is an uncertainty calculation for that collection design, not the spread of individual readings and not a complete confidence interval for an unknown true value.
Now spend the same twelve-reading budget on one day. Its mean has variance 8+4/12=8.333, or SD 2.887. Adding repetitions reduces the within-day term, but it leaves the shared day effect untouched. Treating all twelve rows as independent readings would erase precisely the component the study is intended to detect.
These examples do not prove that three days are enough. The day component has very limited information here. Estimate uncertainty in the components before using the budget for a consequential acceptance rule. If day effects dominate, collecting more independent days is the relevant design change; rerunning a single day more often has diminishing value.
Handle a negative component without inventing negative variance
For an independent boundary example, suppose within-day variance remains 4 with four readings per day, but the estimated variance of day means is 0.5. The subtraction gives 0.5−4/4=−0.5. Physical variance is not negative. The negative estimate can arise from sampling variation when one noisy estimated component is subtracted from another.
Preserve the raw estimate and method, and state how the reporting convention handles the boundary, for example displaying a zero-constrained day component while retaining the unconstrained result in the audit table. Do not say the negative number proves there is no day effect. NIST’s own discussion warns that a negative component based on limited degrees of freedom need not be reliable evidence about the broader process.
A second boundary is systematic error. A workflow could always read the chart ten units too high and still have tiny within-day and between-day variance. Repeatability measures precision, not accuracy. An independent reference or a known check value is needed to investigate the common offset. Averaging more repetitions does not remove it.
Diagnose the design before pooling observations
Plot readings and day means in collection order. A monotone learning effect, a software update or a changing reference object may contradict exchangeable random days. If that happens, do not relabel the trend as a stable random variance component. Identify a version boundary, model the change explicitly, or collect a new stable run.
For multiple chart objects, keep object identifiers and compare repeatability at each level before pooling. Differences between true chart values are not measurement noise. Similarly, unequal repetition counts require the appropriate design calculation; inserting an arbitrary average J into the balanced formula can distort the subtraction. Missing attempts should retain their status and cause rather than disappear selectively from difficult days.
Use this collection and review sheet:
| Check | Required record | Action on failure |
|---|---|---|
| Same object | archived bytes, level, unit and hash | separate real changes from measurement |
| Independent repetition | fresh execution and settings | remove copied outputs from effective repetitions |
| Nested structure | object/day/run/repetition identifiers | reconstruct hierarchy before estimating |
| Pooling | sums of squares and degrees of freedom | avoid averaging SDs |
| Model adequacy | level, day and order plots | split regimes or revise assumptions |
| Reporting target | single result, day mean or multiday mean | use that target’s variance formula |
The final note should retain both estimates and the collection schedule. Specify what would change the conclusion: new independent days, evidence of a shared software shift, a reference-object revision, or a validation showing persistent bias. A more precise same-day average does not erase unresolved between-day evidence, and good precision does not by itself certify the research quantity as correct.
Audit pooling with unequal information
Before applying a balanced day-effect formula, test the pooling calculation separately. Suppose two otherwise comparable within-group variance estimates are 4 with 3 degrees of freedom and 9 with 7 degrees of freedom. Their pooled variance is (3×4+7×9)/(3+7)=7.5, so the pooled SD is about 2.739. Averaging SD 2 and 3 gives 2.5, whose square 6.25 is a different quantity. An unweighted average of the variances would also ignore their unequal information.
This small arithmetic check does not make an unbalanced multilevel design balanced. It verifies only how compatible within-group sums of squares are combined. For the day component, preserve each day’s actual repetition count and use a method suited to that design. A successful pooling check cannot justify inserting the main example’s J 4 after a day lost measurements. Keep failed attempts, usable counts and the chosen estimation method in the collection record so another reviewer can identify which formula was applicable.
FAQ
Can twelve same-day runs replace three days of checks?
They reduce a different variance term. They do not independently observe three day effects, so the two designs cannot be treated as equivalent.
Should a negative day variance be reported as a real negative quantity?
No. Preserve the unconstrained estimate and any boundary convention. A zero-constrained display is not proof that the population day effect vanishes.