A strategy earns $240 for the year, with $200 coming from one asset. Its report then removes the worst period, redraws a smoother curve, and announces that it has passed robustness testing. Removing a bad period and investigating the largest contributor both change a table. What does each change, and which results still explain the original account?
Before editing another chart, distinguish three questions. How sensitive is the result to meaningful changes? Where did the profit come from? Can those contributions be interpreted as persistent skill? Different checks answer different questions. Combining every table into one “passed” label hides the distinctions that make the investigation useful.
What you will judge
- Separate period and asset contributions from tests of rule components.
- Interpret nearby parameters, adverse conditions, and model exposures with their limitations.
- Save every outcome in a robustness matrix, including additional trials and the evidence still needed.
Write the question before opening the result
A robustness check asks how a conclusion changes under a meaningful change in conditions. It does not require every asset and every subperiod to be profitable. Nor does it mean trying conditions until a stable-looking region finally appears.
You might plan checks across fixed periods, reliance on a few assets, nearby parameter values, and higher costs or delayed execution. CFA Institute's public overview presents scenario and sensitivity analysis as supplements to backtesting.[1] The purpose here is to ask additional questions, not to create an automatic reliability certificate.
Explain why each check matters. A trend mechanism may incur costs in a repeatedly reversing market. A stock-selection rule may depend on an industry environment. If no conceivable outcome could challenge the explanation, the test is unlikely to change your judgment. State a possible failure before finding a story for the observed result.
Break $240 down in two directions
Use the independent C6 synthetic reading exercise below. It neither replaces an earlier report nor supplies a real C2 backtest. One full year is divided into three equal consecutive periods, P1, P2, and P3. The account begins with $10,000, has no external flows or leverage, and excludes personal taxes. All supplied profits and losses already include the same teaching cost treatment.
Fictional assets X, Y, and Z occupy separate subaccounts starting at $4,000, $3,000, and $3,000. No funds move between them. Each cell supplies a dollar change in subaccount equity, including realized and unrealized profit or loss. These are dollar contributions, not periodic returns that can be added. No actual strategy or transaction-price record is supplied.
| Source | P1 profit/loss | P2 profit/loss | P3 profit/loss | Full year |
|---|---|---|---|---|
| X | 100 | -20 | 120 | 200 |
| Y | 30 | 0 | 30 | 60 |
| Z | -10 | -10 | 0 | -20 |
| Total | 120 | -30 | 150 | 240 |
| Preselected benchmark account | 80 | -20 | 90 | 150 |
Adding down the columns gives $120, negative $30, and $150; adding those gives $240. Ending equity is $10,240, a 2.4% annual net return. The benchmark also starts with $10,000 and matches the year, currency, and net-cost basis. Its $150 supplied profit is 1.5%, leaving the strategy ahead by 0.9 percentage points.
Across assets, X contributes $200, or approximately 83.33% of the $240 net profit. That denominator is profit, not invested capital or risk. Offsetting gains and losses can make such a contribution share exceed 100%. It is not a portfolio weight and does not establish that X generated 83.33% of total risk.
Keep the losing period visible
In P2 the strategy loses $30 while the benchmark loses $20. Investigate exposures, trading, cost compliance, and whether the proposed mechanism anticipated such conditions. A profitable year does not erase P2. Conversely, one losing period does not by itself refute the strategy.
P3's $150 contribution does not entitle it to greater evidential weight. Calling that period “normal” only after observing its profit, then excluding P2 as abnormal, redraws the evidence boundary using the outcome. Preserve the full comparison and the reason each environment is relevant.
Period analysis begins as diagnosis. Retain the original development, validation, and test roles. A viewed subperiod cannot become unknown again through a new label. An environment classification needs an advance or independent basis, with the observations outside that classification displayed too.
Remove the largest winner without changing the funding
Follow a predeclared diagnostic: remove the year's largest positive contributor. Once X is identified, replace its $4,000 subaccount with cash for the entire year, earning zero. Leave Y, Z, and all other conditions unchanged. Do not transfer that $4,000 to the remaining assets.
The remaining profit is 60 minus 20, or $40. Relative to the original $10,000 account, return is 0.4%. The year relies heavily on X's contribution. This does not show that you could have identified the asset in advance, or that a trend mechanism must profit on every asset separately.
If the question remains how replacing X with cash compares with the original opportunity cost, keep the preselected benchmark. The cash version returns 0.4% against its 1.5%, trailing by 1.1 percentage points. That comparison remains meaningful even if the benchmark contains X: giving up X is precisely the change being diagnosed. Do not select an easier benchmark just to let the modified version win.
If instead you ask whether the remaining rule works within an opportunity set excluding X, you need a corresponding comparison without X. The table supplies no asset breakdown for the benchmark, so its 1.5% cannot reveal that new comparison's return. The first question measures the cost relative to the original alternative; the second investigates selection within a narrower universe. Register the questions and benchmarks separately rather than combining them into one conclusion.
Scaling Y and Z up to invest all $10,000 would add a different funding and risk policy. It cannot be reported as the same removal check. The winner is identified using hindsight: a useful diagnostic of concentration, not an alternative that the original strategy could necessarily implement beforehand.
Removing a rule component is another comparison, often called ablation. For example, remove an extra confirmation condition while preserving the remaining definitions, then regenerate trades, exposures, costs, and the account. Deleting profitable trades from an existing table is not equivalent to rerunning the modified rule.
Nearby parameters and rule comparisons
The next table supplies prearranged synthetic results for reading practice. These net dollar outcomes are teaching inputs, not results the author obtained from real prices. This independent baseline has a smoothing window of ten and an extra confirmation condition; do not silently identify it with C2-A. Windows nine and eleven change only the window. Ablation removes only confirmation, with each version's other frozen specifications and the cost method preserved.
| Check | P1 net profit/loss | P2 net profit/loss | P3 net profit/loss | Full year |
|---|---|---|---|---|
| Baseline window 10 | 120 | -30 | 150 | 240 |
| Nearby window 9 | 100 | -25 | 135 | 210 |
| Nearby window 11 | 95 | -28 | 138 | 205 |
| Remove extra confirmation | 140 | -110 | 160 | 190 |
| Increase cost assumption only | 90 | -45 | 125 | 170 |
Both nearby windows remain profitable for the year. This reduces concern that favorable results exist only at one exact setting. It does not prove the absence of overfitting. The same history can carry shared data errors and exposures, and three settings cannot represent all parameters or environments.
Without confirmation, annual profit is $190, but P2 loses $110 instead of $30. Confirmation may affect exposures, trading, or costs. Inspect intermediate records before claiming it caused better risk control. Removing a condition can alter positions; it is not necessarily just one fewer logical check on otherwise identical holdings.
Higher costs leave $170 rather than $240, showing cost sensitivity. This result does not replace G44's transaction-by-transaction reconciliation or determine real capacity. Each row should point to a specific conclusion and the next evidence needed, rather than contribute a generic green mark.
Benchmark difference, model explanation, and skill
The annual difference between 2.4% and the benchmark's 1.5% is 0.9 percentage points. Explaining its source requires more than that subtraction. Use a given single-factor teaching model: annual risk-free return is 0.5%, the market reference is the same 1.5% benchmark, and strategy beta is assumed to be 1.6.
The model reference is 0.5% plus 1.6 times the difference between 1.5% and 0.5%, or 2.1%. The strategy exceeds it by 0.3 percentage points. This is a single-period residual, with persistent alpha and period-specific error not separated. Beta is an input, not an estimate obtained from this one year.
Now use another predeclared model that additionally includes style exposure of 0.2 and a same-year style-factor return of 2%. The added explanatory component is 0.4 percentage points. The model reference becomes 2.5%, and the residual becomes negative 0.1 percentage points. Both models are synthetic conditions; the exercise does not establish which better represents real information.
Model choice can change the size and even sign of unexplained return. CFA Institute distinguishes performance measurement, attribution, and appraisal, with attention to what each permits you to infer.[2] A model explaining correlated returns does not prove risk is the sole cause. A positive remainder does not independently demonstrate skill.
Real research also needs holdings, exposures across time, and estimation uncertainty. A factor that makes alpha disappear is not automatically the final truth. Nor should you keep switching models until a preferred positive alpha remains. The explanation process itself has choices that belong in the research history.
Robustness work can become another search
Lin adds a date range, removes two assets, changes a normalization, and introduces a filter. Each action can sound diligent. If she preserves only favorable outcomes, however, the sequence remains a search for a report that passes.
Bailey and coauthors study how searching many strategy configurations on finite history can produce attractive false discoveries.[3] You need not calculate their statistical method to adopt a practical discipline: number every additional check and retain its result and influence on later changes.
Preregistration does not forbid a justified new diagnostic after a problem appears. Identify it as added afterward. If it then selects a rule, the historical information has entered selection and another evaluation is needed. Do not rewrite the final version as though it had been your only original plan.
Explain what each result can establish
Why does a remaining $40 profit after removing X fail to prove independence from every single asset? Does positive annual profit of $190 without confirmation establish that confirmation has no value? Write what the checks answer and what remains unknown.
Reference answers · reveal
Answer guide: Only removal of the largest contributor was checked, and profit fell from $240 to $40. That indicates substantial reliance rather than proving universal independence. For confirmation, inspect P2, exposures, trading, and costs as well as the annual sign. A positive endpoint neither identifies the rule's role nor proves causation.
Same profit, greater concentration
In another independent synthetic version, annual contributions are $300 from X, $20 from Y, and negative $80 from Z. Starting capital is unchanged. Calculate total return, X's share of net profit, and the result after replacing X with zero-return cash without scaling the others. Is a contribution above 100% necessarily a table error?
Reference answers · reveal
Answer guide: Total profit is still $240 and return still 2.4%. X contributes 300 divided by 240, or 125% of net profit. Offsetting losses explain the figure; it is not a portfolio weight above 100%. Without X, the other contributions total negative $60, or negative 0.6% of the original account. Identical endpoints conceal greater concentration, requiring more contribution and exposure investigation.
Save a robustness and explanation matrix
For each check, record purpose, advance or after-the-fact status, fixed conditions, the one change, data version, net result, and risk changes. Add the supported conclusion, alternative explanations, and next material to inspect. Keep failed and unrun checks as separate visible statuses rather than recording planned work as passed.
Include period and asset breakdowns, largest-contributor removal, ablation, neighboring parameters, and adverse implementation. If you have only read this supplied exercise, label the output accordingly. Missing real C2 prices, account records, and versions cannot support a claim that the actual strategy passed the matrix.
Display all failures and unrun rows when checking the final conclusion. Then inspect whether benchmark-relative return, model explanation, and skill have been conflated. The matrix should narrow uncertainty and organize investigation. Its purpose is not to make every cell favorable.
Keep diagnosis and explanation distinct
Show all planned changes and retain losing periods and failed checks. Removing a winner or ablating a rule provides diagnostic information, not an automatically implementable alternative or causal proof. Similar nearby results cannot replace new evaluation information.
Trace contributions to common exposures and model limitations. One period's residual is not skill, and an attractive total can be concentrated. As checks accumulate, the trial record becomes more important. Preserving outcomes and revision reasons lets robustness work improve the investigation instead of becoming another selection exercise.
References
- CFA Institute: Backtesting & Simulation, 2026 public overview. Scenario and sensitivity analysis alongside historical backtesting; not the paid full reading.
- CFA Institute: Portfolio Performance Evaluation, 2026 public overview. Measurement, attribution, appraisal, and inference limits.
- Bailey et al.: The Probability of Backtest Overfitting, February 2015 author version. Configuration searches on finite history; no PBO calculation or strategy acceptance is claimed here.
Sources inspected October 30, 2026. AI-assisted draft; independent AI full-text and bilingual review is complete. Human financial review and target-reader testing have not been performed. All numerical tables are synthetic reading inputs, not C2 or actual investment performance.