A strategy beats its benchmark in nine out of ten observations. Lin is ready to investigate it further. Then the researcher mentions that it was the best of 100 versions. Should the nine wins carry the same evidential weight as nine wins from a rule specified before any results were seen?
The account may reconcile and every input may have been timely. Yet repeated searching gives the researcher more opportunities to find an attractive accident. Understanding that selection process requires the records of attempted versions, not just the winner's curve.
What you will be able to do
- Calculate a simple multiple-trial probability and explain its assumptions.
- Distinguish an exploratory winner from evidence obtained after a rule was fixed.
- Save unsuccessful, blocked, and abandoned attempts alongside a frozen evaluation plan.
Overfitting can begin with a simple rule
Overfitting occurs when a choice adapts to peculiarities of observed data without sufficient evidence that it will generalize. A complicated model can do this, but so can a simple threshold selected after many trials, a favorable date range, deletion of a bad period, or a change in the asset universe.
Exploration is part of research. The problem arises when a result chosen through exploration is presented as if the rule had been fixed independently of those outcomes. Repeated use of evaluation data to guide choices weakens its role as a new test.1
G41 examined information entering a calculation before it was available. Here, even chronologically correct calculations can be selected after seeing their outcomes. A clean implementation cannot reconstruct a missing search history.
A model with no skilled candidates still produces winners
Use this independent synthetic C4 error-laboratory supplement. It is a probability mechanism, not an empirical result for SPY or the book's C2 strategy.
Each candidate has ten independent observations. In each observation, its gross active return relative to a benchmark is either +1 percentage point or −1 percentage point, each with probability one-half. Candidates are also independent of one another. There are no costs in this model, and every candidate's true expected active return is zero.
The display rule is “show candidates with at least nine positive observations out of ten.” That is a selection criterion, not a definition of a qualified strategy.
There are 2 to the tenth power, or 1,024, equally likely sequences for one candidate. Exactly nine positives can occur in ten positions for the single negative; ten positives form one more sequence. Thus 11 sequences satisfy the display rule:
For one candidate fixed in advance, that particular event is relatively uncommon under the supplied model. Now try 100 independent candidates and display any that qualify. The chance that none qualifies is the failure probability for one, multiplied by itself 100 times. The complementary probability is:
An impressive-looking record is now a common batch outcome even though the mechanism contains no skill. The number does not mean there is a 66.04% probability that a chosen real strategy is useless. We assumed zero expected active return for every candidate at the outset. We calculated the probability of finding at least one qualifying display within that model, not a posterior probability about an actual strategy.
Real candidates often share data, signals, and parameters, so they are not independent. Do not insert an observed count of correlated strategies into this formula and treat the result as an empirical false-discovery probability. The example establishes why the search process matters, not an automatic correction for every research project.
Preserve the selection before showing the next observations
Here is a separate four-observation example. Every sign still represents an active return of +1 or −1 percentage point. The table is the complete initial comparison among three candidates.
| Candidate | Initial four observations | Positive count | Mean active return |
|---|---|---|---|
| A | +1, −1, +1, −1 pp | 2/4 | 0 pp |
| B | +1, +1, +1, −1 pp | 3/4 | +0.5 pp |
| C | −1, −1, +1, −1 pp | 1/4 | −0.5 pp |
Pause here and save all three rows, the criterion, and the decision to select B before reading the later observations. If you have already seen the rest, label your work a reconstruction of the teaching mechanism. It is no longer an unseen prospective test for you.
B has a 75% positive-observation rate and the best initial mean. Reporting only B hides how it was selected. This is selection among research candidates; it differs from historical-universe survivorship bias, where a sample omits assets that later disappeared.
Now reveal B's next four independent observations: −1, +1, −1, +1 percentage points. Their mean active return is zero. That outcome is consistent with the model's lack of skill. It does not permit you to inspect A and C's later outcomes, switch to the more attractive one, and still describe the original comparison as an untouched test.
The key distinction is between “B was specified before these evaluation outcomes” and “B was selected from several alternatives using these outcomes.” The same visible record can have a different research history.
Count choices honestly, including incomplete attempts
A trial can involve a parameter, threshold, holding period, asset universe, cost model, date range, preprocessing method, or evaluation metric. A changed question can create another hypothesis even if the program makes only one more call. Three software runs may conceal many manual choices. Conversely, an exact rerun of unchanged inputs and rules is not automatically a new independent hypothesis.
Suppose a momentum rule looks unsuccessful and the researcher flips its direction to mean reversion. That reversal is a new hypothesis. Preserve the original failure, when the reversal was proposed, and the evaluation plan for the revised rule. The reversal does not prove that the original rule succeeded, and the already inspected history does not become fresh evidence for the opposite direction.
Do not erase attempts merely because no complete report was produced. The following statuses describe a teaching research log, not actual software executions:
| Version | Observed record | Status to preserve |
|---|---|---|
| A | All four observations; mean 0 pp | Completed |
| B | All four; mean +0.5 pp | Selected from A, B, C |
| C | All four; mean −0.5 pp | Completed, unfavorable |
| D | Two required inputs missing | Blocked; no complete result |
| E | Stopped after seeing two negative observations | Abandoned; no full four-observation evaluation |
D and E do not disappear from the research history. Their results are also not comparable with complete four-observation records. Distinguish a data blockage, a rule conflict, a cost problem, and abandonment after an unfavorable outcome. Preserve the actual reason and what had already been seen.
This does not mean you can put five fully comparable independent trials into a statistical formula. Incomplete histories and correlated candidates require their own treatment. An honest log can expose what is unknown without manufacturing a precise trial count. If reconstructed from memory, label uncertain entries as retrospective estimates rather than claiming exact timestamps or completeness.
For each version, save rules, data version, period divisions, criterion, cost assumptions, run or viewing time, observed outcomes, and the reason for a later change. An unfavorable result is information about the hypothesis, not an administrative error to remove.
Freeze the next evaluation before observing its results
Write the main question, main rule, finite candidate set, allowed cost scenarios, metrics, stopping conditions, and permitted occasions for inspecting results. Save that plan before the relevant outcomes are seen. New exploration can then start a new recorded phase without overwriting the old one.
In the book's C2 coursework, C2-B allows one preregistered variation. Additional attempts belong in the log; a failed main rule cannot quietly be replaced with the best later winner. Development or validation data already used for choices do not become a new holdout when renamed.
C2 is a separate proposed real SPY study for 2007–2024, with 2006 warm-up data. Preserve G37's frozen strategy specification when comparing versions rather than silently changing its signal, execution, cost, or missing-data rules.
The periods 2007–2014, 2015–2019, and 2020–2024 serve development, validation, and a historical teaching holdout. The final period is already known history to the author; it is not future evidence. Actual forward observation begins after a rule is frozen and still needs an adequate evaluation period. The real study's R4 gate remains blocked and T09 has not started. This chapter supplies no validated-strategy result and does not substitute a toy probability exercise for those missing steps.
Advanced methods need more than the winning curve
Two research methods help formalize parts of the problem. You are not required to calculate them for this chapter.
The probability of backtest overfitting, or PBO, is the probability that the in-sample-optimal selection ranks below the median out of sample under the specified candidates and partitions.2 It is not the probability that you will lose money in the future or a direct probability that a real strategy has no skill.
The paper's combinatorially symmetric cross-validation, CSCV, uses a comparable candidate-performance matrix and a specified selection procedure. Its partitions do not automatically become independent observations of the actual future. Omitted candidates, information leakage, and changing market conditions limit what the estimate can establish. A single winning curve is insufficient to reconstruct the necessary comparison.
The deflated Sharpe ratio, or DSR, addresses inflation associated with selection and non-normal returns. Its calculation needs more than the winning Sharpe value: relevant inputs include sample length, the variability of candidate Sharpe estimates, skewness, kurtosis, and an estimate of the effective number of independent trials.3 Correlated candidates are not automatically that many independent trials.
Neither method repairs an incorrect ledger, retrieves missing trial logs, supplies an economic mechanism, or guarantees future profits. We calculate no PBO or DSR estimate in these exercises. Naming an advanced method is not evidence that it was implemented or that its assumptions were satisfied.
Explain selection, then change the experiment
Explanation. Why should “B was selected from A, B, and C” accompany B's initial 75% positive rate?
Reference answers · reveal
Answer. That record helped determine which candidate was shown. The omitted alternatives reveal the selection opportunity. It is weaker evidence of generalization than a matching outcome from a candidate fixed before those evaluation observations were known.
Application. Can E be deleted because it never reached four observations?
Answer. No. Preserve the partial observations, abandonment point, and reason. Do not call it a completed four-observation result, but do not hide the decision made after seeing losses either.
Transfer. Each candidate now has four independent observations of ±1 percentage point, each sign equally likely. Candidates are independent, all have true expected active return zero, and the display rule requires at least three positives. Calculate the qualifying probability for one candidate and for at least one of three candidates.
Answer. There are 16 equally likely sequences. Four have exactly three positives and one has four, giving 5/16 = 31.25% for one candidate. For three independent candidates, the chance of at least one qualifying record is 1 − (11/16)³ = 67.5048828125%, approximately 67.50%. These are probabilities under the supplied mechanism, not measured false-positive rates for correlated real strategies or a direct verdict about an actual winner. Retaining only its 75% positive rate would omit the search that generated it.
Save the attempts and the frozen plan
Retain completed, unfavorable, blocked, and abandoned versions. Include what results were visible when each choice was made, changes and reasons, source materials, and uncertainty about unrecorded historical exploration. Do not invent an exact count to make the file look complete.
Alongside the log, save a dated evaluation plan with the question, main rule, finite comparisons, data and time partitions, costs, metrics, permitted inspection points, and stop or restart conditions. Preserve the version saved before later outcomes; subsequent changes receive a new version.
Hide the best curve and ask whether another reader could reconstruct how it was chosen and find the failed or incomplete attempts. If not, a performance number or isolated probability cannot supply the missing evidence. Research can produce useful failures; the record must allow them to remain visible.
References
- scikit-learn: Cross-validation—Evaluating Estimator Performance. Training-data evaluation and repeated test-informed choices can overfit.
- Bailey, Borwein, López de Prado, and Zhu: The Probability of Backtest Overfitting, 2015 version, Definitions 2.1–2.2 and the CSCV framework. The numerical probability exercises here use separate synthetic mechanisms.
- Bailey and López de Prado: The Deflated Sharpe Ratio, 2014 version, selection bias, non-normality, and effective independent trials. No DSR estimate is produced here.
Editorial status
AI-assisted draft synchronized to the accepted second-round Chinese source. Probability and candidate examples are synthetic; the real C2 study remains incomplete. See the author review. A separate AI reviewer has completed an independent full-text review. Human financial review and target-reader testing have not been performed and are not scheduled for this edition.