Specification sensitivity: report the choices that move results
Specification sensitivity asks whether an analytical conclusion depends on reasonable choices about the dataset and method. Vary sample rules, joins, weighting and outlier treatment separately, then identify the choices that actually move the result. This is different from changing an economic forecast assumption such as a company’s future sales growth.
Start with a reproducible baseline
Write down the baseline universe, period, observation key, source versions, join relationships, missing-value rule, weighting basis and treatment of extreme values. Save the code or worksheet that produces the result. If the baseline is only an AI-generated paragraph, you cannot tell whether a later difference came from a deliberate specification change or inconsistent calculations.
The UK government’s AQuA Book guidance discusses quality assurance for analysis, including communicating uncertainty and limitations. It supports a disciplined approach to analytical choices. The small matrix below is an original research example, not an official scoring method or evidence that a particular investment conclusion is robust.
A hypothetical specification matrix
Assume a cross-company study concludes that operating margins improved. The baseline change is positive 1.2 percentage points. Four alternative specifications produce different results. These numbers are invented to demonstrate reporting; they do not describe any market or actual portfolio.
| Specification | Change in margin | What the change tests |
|---|---|---|
| Baseline, revenue-weighted full panel | +1.2 points | Published baseline |
| Continuing issuers only | -0.3 points | Sensitivity to entry and exit |
| Equal issuer weights | +0.2 points | Influence of company size |
| Strictly matched operating definitions | +0.8 points | Accounting comparability |
| Exclude one acquisition-driven observation | +0.1 points | Concentration in an unusual event |
The headline is sensitive to panel composition: the continuing group declines while the full panel improves. That does not make the baseline calculation false. It changes the defensible interpretation from broad company-level improvement to an aggregate result partly driven by who is included and how much weight they receive.
Change one decision at a time first
Single-choice comparisons help identify mechanisms. If you simultaneously remove entrants, switch weights and cap extremes, you cannot attribute the change to one decision. After inspecting individual effects, test a few justified combinations where interactions matter. Keep a record of every tested specification instead of publishing only favorable variants.
Distinguish corrections from alternatives. A many-to-many join that triple-counts revenue is a defect to fix, not a reasonable specification to include as an equal member of a robustness range. Similarly, mixing incompatible units is not analytical discretion. Sensitivity analysis should explore defensible choices after known errors are corrected.
Define materiality before choosing a headline
Decide what would materially change the conclusion: a sign reversal, loss of most of the magnitude, concentration in one observation or a change in coverage. A result can stay positive yet become economically negligible. Conversely, a small sign change around zero may reveal that a strong directional statement was never warranted.
Show the relevant sample size and covered exposure for each specification. A strict-definition subset may be more comparable but represent much less of the universe. Its result should not automatically replace the broader estimate without discussing that tradeoff. If alternatives answer different questions, label those questions rather than collapsing everything into one uncertainty interval.
Report the decision-sensitive conclusion
The output should explain which choices matter, why they matter and which evidence could resolve the uncertainty. In the example, examining entrants and the acquisition event is more useful than averaging all five estimates into a new unexplained number. A range across specifications is not a statistical confidence interval and carries no automatic probability interpretation.
Save the specification registry with the final memo. AI can help enumerate choices and draft comparisons, but the analyst should inspect the calculations and select a conclusion consistent with the full matrix. A statement that remains carefully conditional is more informative than a claim of robustness based only on unchanged wording.
FAQ
Should every possible specification be tested?
No. Focus on defensible choices with plausible influence on the conclusion, document why they were selected and avoid arbitrary combinations that no longer answer the research question.
Is the specification range a confidence interval?
No. It describes variation across chosen methods and populations. It does not quantify sampling probability without an additional statistical model.