The last five years are labeled “final test.” The first result disappoints, so the researcher changes the signal. The third version looks good. The report still calls it excellent out-of-sample performance because the program never passed those five years into its training function. Did the information nevertheless teach the researcher which rule to choose?
Learning can pass through the researcher as well as the code. If a result changes the rules, parameters, or research question, it has helped select the reported version. To judge how much independent information a test retains, inspect both its dates and the history of who saw what and what changed afterward.
What you will organize
- Assign development, validation, and final testing different jobs, preserving every viewing and revision.
- Draw rolling or expanding windows that move forward through time, checking when training information is complete.
- Address overlapping labels, a previously viewed holdout, and future observations in a T06 window plan and viewing log.
This chapter teaches design and review. It specifies no universally best split and does not run the real C2 historical strategy. The monthly examples below are separate synthetic C4 exercises about timing, not additional investment performance.
Give each interval a job
Development data help you understand the material, implement rules, choose features, and estimate parameters. Validation data help select among declared candidates using declared evaluation methods. Because validation participates in selection, it cannot simultaneously provide the final independent assessment of that selection.
Use the final test as agreed after locking the rules and evaluation method. Its job includes revealing unfavorable evidence. It is not an exam that every project must eventually pass. If a genuine implementation error requires repair, record the information already seen, the repair, and the resulting limitation on independence.
The scikit-learn leakage guidance excludes test information from model choices as well as fitting and requires preprocessing to learn from training data.[1] In a personal project, this means an AI assistant should not keep searching for better rules using the very results you intend to call final.
“Out of sample” describes information's relationship to a particular selection process. It is not a permanent property of a market-data row. History used for testing in one project can become development material in another. Record that change of role; do not report the later project as though the earlier viewing never happened.
Preserve the C2 boundaries
C2-A assigns 2007–2014 to understanding and developing the rules, 2015–2019 to validation, and 2020–2024 to a frozen teaching holdout. Data beginning in 2006 provide warm-up. Keep those years instead of extending the exercise to the present because a later market looks more favorable.
The three intervals contain eight, five, and five calendar years. With one valid month-end observation per calendar month, that would mean 96, 60, and 60 observations. Actual counts require checking the available data and trading calendar. Calendar arithmetic does not prove that the real dataset is complete.
The years 2020–2024 are also already historical teaching material. They are not a future the author has never encountered. The correct description is a frozen teaching holdout. Readers who have inspected its curve or changed a parameter because of it should record the influence. Naming a folder “test” cannot make familiar information unknown again.
The fixed C2 rule need not estimate machine-learning parameters at all. A researcher can still choose the asset, moving-average window, starting date, ending date, and cost assumptions during development. Register the entire selection process rather than concluding that a strategy without a training function cannot overfit.
Why not shuffle every month?
If the task is to observe the past and make the next decision, randomly placing later months in training may expose the model to a later price environment. Dependence, changing conditions, and outcomes spanning several months can make shuffled rows a poor representation of the intended process.
TimeSeriesSplit offers chronological partitions and notes the equal-spacing condition for comparable durations across its folds.[2] This does not establish that random splitting is wrong for every statistical task. The split must represent the decision you are studying.
Dates alone are insufficient. Standardization, missing-value treatment, feature selection, and model estimation must all respect the information allowed in that training window. Apply an already determined transformation to the test segment; do not use its later mean to redefine the training scale.
To review a proposed normalization, ask where its mean and spread came from. If the answer is “the whole file,” chronological train and test labels have not solved the problem. The permitted information boundary applies to intermediate calculations as well as the final model.
Walk forward with an explicit updating rule
Take 18 consecutive, equally long synthetic months, M1 through M18. Initially assume that all information required for calibration is available by each calibration endpoint. We have not yet introduced a label requiring future outcomes. Recalibrate using the most recent six months, then assess the next three without changing the method in response to those three results.
| Round | Latest six months available for calibration | Following assessment months | Expanding-window alternative |
|---|---|---|---|
| 1 | M1–M6 | M7–M9 | M1–M6 |
| 2 | M4–M9 | M10–M12 | M1–M9 |
| 3 | M7–M12 | M13–M15 | M1–M12 |
| 4 | M10–M15 | M16–M18 | M1–M15 |
The fixed-length calibration window rolls forward. The alternative expands by retaining more known history. Both can be reasonable designs, but they treat older information differently. Choose and explain the method before comparing outcomes rather than preserving whichever subsequently looks best.
The four assessment blocks cover 12 months: each of M7 through M18 appears once. Overlap among training windows creates no extra assessment months. Counting recurring training rows as new out-of-sample observations would overstate the record's length.
CFA Institute's public backtesting overview describes rolling calibration and subsequent performance tracking, while retaining look-ahead and structural-change limitations.[3] A moving window makes the allowed learning process explicit; it does not eliminate the risks in the underlying information.
By round two, M7–M9 have occurred and may enter the next calibration according to the predeclared rolling or expanding process. That is different from rewriting the first round after learning its result. Lock the updating schedule and selection method before evaluating them.
The frozen object can be an updating procedure rather than coefficients that never change. Reestimating every three months from the latest six, as agreed in advance, is one dynamic rule. Changing to nine months because a loss appeared creates a new version. Log that decision and what prompted it.
When connecting performance across windows, also connect the account. Holdings, cash, and rebalancing costs do not reset for free at each boundary. If the assessment measures prediction error alone, label it accordingly. A smooth series of forecast scores is not yet a continuous, implementable account history.
A row can have a date before its answer exists
Now change one condition. Each month-end feature predicts the compounded return over the following three months. The answer for row M6 is therefore complete only at the end of M9. You want to calibrate after M6 and begin assessment in M7. Can training include every labeled row from M1 through M6?
No. M4's outcome uses M5, M6, and M7. M5 needs M6, M7, and M8; M6 needs M7, M8, and M9. All three extend beyond calibration. Under these stated availability assumptions, the latest complete label is M3, built from the outcomes in M4–M6.
Exclude the immature M4–M6 labels or redesign the calibration and evaluation times. A gap or removal of overlapping labels must follow their actual end and availability times. It is not a universal instruction to leave three empty months for every strategy. A software gap setting cannot determine what an outcome means.[2]
Overlapping historical features are a different issue. An M7 feature built only from information already available in M1–M6 is not automatically leakage because it crosses a training boundary. Check that it borrows no future values and uses no transformation estimated afterward. A shared lookback and an unavailable future answer are different mechanisms.
What to do after seeing the holdout
Suppose Lin views the 2020–2024 results and changes a ten-period average to twelve periods. That interval has entered selection. Dividing it into 2020–2022 and 2023–2024, or renaming the latter “real final test,” cannot erase the information she saw.
Keep the original version and viewing date. Describe what the result influenced and treat it as used research material. Further analysis can still produce exploratory insight. It cannot honestly claim the same independence as a first encounter with genuinely isolated information.
A real bug repair also belongs in this history. The correction may be necessary and well justified, but record how the error was found, which outputs had been seen, and what the rerun establishes. Hiding the earlier version conceals part of the actual search.
If other information truly has not informed this selection or been disclosed indirectly in summaries, document its isolation and why it was chosen. When that cannot be established, leave the uncertainty visible. Repeatedly searching for a favorable period until a test passes adds selection rather than independence.
Forward observation begins after freezing
Correct historical ordering and a researcher who has genuinely not seen the outcome are separate conditions. A forward record starts when the rules and process are frozen, then newly arriving information enters in sequence with timestamps preserved.
Paper observation retains simulated-fill limitations. Small real trades add order, account, and operating questions. Neither passes because a few days show a profit. Evaluate record length, trade count, market conditions, and differences from the historical study separately.
Conditions can change. Predeclared monitoring rules can trigger a pause, review, or new project version. Keep the frozen rule's forward record separate from development of its replacement. A record adjusted along the way cannot be described as one unchanged strategy.
Check the information boundary yourself
Explain why a final segment loses independence when the researcher changes a parameter from its results, even without a training-function call. Then apply the window table: may round three use the mean of M13–M15 to standardize M7–M12? Write the permitted information first.
Reference answers · reveal
Answer guide: Human selection is an information route. In the second question, the mean comes from the future assessment segment and therefore crosses this round's boundary. Estimate the transformation from permitted training information and apply that same transformation to subsequent values.
A project whose entire chart was viewed
An independent synthetic project originally assigned M1–M12 to development, M13–M18 to validation, and M19–M24 to final testing. Its author has seen the full M1–M30 chart and changed the rule because M19–M24 disappointed. Can M25–M30 automatically become an unknown holdout by relabeling it?
Reference answers · reveal
Answer guide: No. That segment appeared in the viewed chart and may have affected asset, rule, or window choices. Log all 30 months and the reason for revision. Describe the new analysis as further exploration of seen history, then arrange genuinely isolated or subsequently arriving evaluation information.
Now calibrate after M12 using the same following-three-month label definition. Which of M10, M11, and M12 can enter with a complete label? None. The latest complete label is M9, using M10–M12 outcomes. Removing only the last row would leave unavailable answers in training.
Save the T06 window plan and viewing log
Draw dated boundaries for development, validation, and final holdout. Add warm-up, feature lookbacks, label endpoints, actual information availability, and the earliest permitted execution time. For rolling or expanding designs, fill each calibration and assessment interval and the account-continuity policy.
Log rule and data versions, who or which tool saw each interval, when, for what purpose, what information emerged, and whether anything changed. Preserve failed and abandoned trials. If AI sees a result and recommends a revision, record that exposure even when you never personally open the chart.
Choose one historical decision and trace backward. Were the required features and labels available? When was its procedure locked? Did assessment results feed into selection? Mark missing evidence as unchecked. A completed-looking checklist cannot replace the underlying timestamps and versions.
Independent evidence needs an honest history
Assign intervals their jobs, restrict learning and preprocessing to permitted information, then inspect label maturity and execution order. Rolling and expanding windows describe updating. They do not automatically remove dependence, repeated viewing, or changing environments.
Log what you saw and version what you changed. Familiar history can continue to teach you, but renaming it cannot make it unknown. Keeping exploration and forward observation distinct makes the next result's evidential contribution easier to judge.
References
- scikit-learn: Common Pitfalls—Data Leakage. Model selection and preprocessing boundaries; no library use is required here.
- scikit-learn: TimeSeriesSplit. Chronological splits, expanding training sets, and the sample-count gap; not an automatic audit of business timing.
- CFA Institute: Backtesting & Simulation, 2026 public overview. Rolling evaluation and its limitations; the paid reading was not inspected.
Sources inspected October 30, 2026. AI-assisted draft; independent AI full-text and bilingual review is complete. Human financial review and target-reader testing have not been performed. The window and label exercises are synthetic; no real C2 strategy was run and no historical teaching holdout is presented as unknown future information.