Keep an honest log when ETF parameters improve
A higher backtest return after parameter changes is a new observation, not automatic evidence of a better method. Once you have looked at an interval to select a rule, that interval is part of the exploration.
Freeze the question before the grid
S06’s first scope is sensitivity for the supported rule families, not open-ended strategy invention. Specify the full combination set, data version, costs, split rule and comparison interval before reviewing outcomes. A method’s availability still depends on its published verification status.
| Experiment field | Teaching record | Why retain it |
|---|---|---|
| Family | Simple momentum | Avoid mixing different strategies |
| Windows | 42, 63, 126 trading intervals | Show all prespecified variants |
| Costs | Named scenarios fixed before results | No selective cost reporting |
| Data | Fixed assets, version and digest | Same input meaning |
| Comparison | Common post-warm-up interval | Equal opportunity to earn |
| Split | Frozen date and selection rule | Distinguish exploration from validation |
The table describes CoTrading’s research protocol, not a regulator’s requirement. Bailey and coauthors’ The Probability of Backtest Overfitting examines selection across backtests. Our simple log does not compute that paper’s overfitting probability; it preserves the evidence needed to see what was tried.
Worked sensitivity reading
Consider invented common-period returns of 3%, 8% and -2% for the three windows. Selecting only 8% hides fragility. Report all three, their costs, drawdowns and failed attempts. These invented outcomes are not generated market results or evidence that one window is preferable.
The 126-interval run needs more history than the 42-interval run. Keep each full result but also calculate a shared post-warm-up interval. Do not compare cumulative returns starting on different dates as if only the parameter changed.
Preserve failed runs
A missing-data error is not a zero return and must not disappear from the matrix. Record run ID, rule digest, data digest, status, error and artifacts. If a high-cost scenario was accidentally skipped, the comparison is incomplete until rerun.
Keep the held-out boundary
If you inspect held-out results and tune again, label the new work exploratory. A genuinely untouched validation segment requires rules and selection frozen before viewing it. Repeatedly changing the cutoff does not create independence.
When to stop
The four-day sample cannot test any of these momentum windows. Return insufficient history rather than fabricate a grid. Robustness does not certify future profits, remove survivorship bias or transform a fixed-asset experiment into whole-market selection.
FAQ
Is a wide parameter grid independent validation?
No. It is exploration unless a separate untouched evaluation was frozen in advance.
May failed runs be removed from the summary?
Keep their status and cause; dropping them changes the evidence set.