Repeated checks: set a stopping rule before looking again
Repeatedly inspecting a research evaluation and stopping at the first attractive result changes the evidence you are reporting. Define the review schedule, decision rule and maximum scope before looking. Operational monitoring can legitimately continue over time, but a fixed-horizon statistical guarantee does not automatically remain valid under an outcome-dependent stopping rule.
Separate learning from a final claim
During development, inspecting failures is useful. The problem arises when the same evolving evaluation is presented as an untouched confirmation, or when only the favorable inspection is reported. A new review date is not a new independent experiment if it reuses earlier observations. Keep the cumulative history and identify which decisions were prespecified.
The NIST sequential sampling discussion illustrates sampling plans with defined accept, reject and continue regions. It concerns lot sampling rather than investment research. Howard and colleagues’ confidence-sequence paper describes time-uniform inference under stated mathematical conditions. Neither source licenses arbitrary repeated testing with an unchanged fixed-sample interpretation.
A hypothetical audit stopped early
Assume a team plans to inspect one hundred extracted disclosure values for material errors. In the invented observation sequence, the first ten contain no errors, the first twenty contain two, the first fifty contain four and the full hundred contain twelve. The cumulative error proportions change as information arrives.
| Inspection point | Cumulative errors | Observed proportion | If only this look were published |
|---|---|---|---|
| 10 values | 0 | 0% | Could suggest a perfect extraction |
| 20 values | 2 | 10% | A different apparent result |
| 50 values | 4 | 8% | Another partial estimate |
| 100 values | 12 | 12% | Planned full-sample description |
Stopping after the first zero-error block and announcing an error-free workflow selects a favorable prefix. The table is illustrative, not a probability calculation for how often this path occurs. It also does not imply that the hundred-value result is exact truth about all future extractions; sampling and label uncertainty remain.
Define the stopping rule with the question
Record the target population, sampling unit, outcome definition, planned inspection times and reason for stopping. A safety-oriented rule might stop operations after a serious error so the team can investigate. That can be a sensible operational decision without making the stopped sample a valid general performance estimate.
A research-quality claim needs an analysis appropriate to its sampling and stopping design. If you intended one final fixed-size evaluation, either follow that design or describe deviations. If ongoing decisions are necessary, choose and validate a sequential method that supports them rather than silently treating every fresh look as a conventional final test.
Do not invent an independence correction
Repeated cumulative looks share observations. Their test statistics are typically dependent, so the formula for several independent tests is not automatically the correct false-alarm calculation. Likewise, changing the nominal significance level by intuition does not establish a controlled procedure. Use a justified method with its assumptions and implementation checked.
Time-uniform intervals, sequential tests and other approaches have specific conditions and goals. They are not interchangeable labels for any dashboard that updates daily. Distinguish confidence intervals, confidence sequences and descriptive proportions in the report. If you have not implemented a sequential guarantee, do not borrow its terminology to describe ordinary repeated estimates.
Preserve all inspections and subsequent changes
Save each look, the data available then, the result and the decision it prompted. Record prompt changes, revised extraction rules and newly excluded cases. Once an evaluation informs those choices, it becomes development feedback. Reserve new cases for a later independent assessment of the revised workflow.
Reporting the complete inspection history helps another researcher distinguish a stable result from selective timing. It also makes an operational stop interpretable: the process may have stopped because of capacity, a calendar deadline or a quality trigger, each with different consequences for what the data can support.
FAQ
Does this mean I should never monitor results continuously?
No. Monitoring is useful. Match the decision and inference to the design, and avoid presenting the first favorable look as independent confirmation.
Can I call every repeated confidence interval time-uniform?
No. That property requires a suitable construction and assumptions. Recomputing a fixed-sample interval does not establish it.