Freeze a stock strategy before opening its holdout test
A holdout test is useful only while its results have not helped choose the candidate being tested. Freeze one stock strategy and its acceptance rule before opening the reserved interval. Preserve the first decision even when it disappoints, and treat later adjustments as new exploratory work rather than overwriting the failed test.
The deliverable is a candidate seal, one opening receipt and an inspectable result or unfinished-run status. Our standard-library demonstration uses invented trade profits and exclusive local files. It does not establish that a real historical interval is unseen, verify CoTrading functionality or prove future profitability. Apply the procedure within one stock market and currency.
Establish the information boundary
Write down the development interval, reserved interval, permitted inputs and who or what has already viewed each result. If the reserved period appeared in a chart, summary metric or model prompt used for selection, it may already be contaminated. Relabel it honestly instead of assuming that a newly created filename restores independence.
The scikit-learn evaluation documentation explains that adjusting hyperparameters against test results can leak test information into selection. Our stock workflow uses a chronological reserved period and a saved candidate; it does not copy a random classifier split as a trading validation design.
Reserve enough data for the proposed question before opening it. Check input availability without revealing outcome-based rankings to the selector. Predeclare how missing observations, insufficient trades and execution errors will be handled. A reserved period with no trustworthy input cannot become useful merely because the candidate has been sealed carefully.
Put the complete candidate into the seal
Include code and configuration references, data snapshot and interval, security universe, calendar, cost assumptions, execution timing and initial-account policy. Record the scoring formula and pass conditions, together with minimum evidence requirements. An evaluator cannot reproduce a rule described only as “reasonable return and low risk.”
Our candidate uses lookback three, a minimum of two completed synthetic trade observations and aggregate net profit at least zero. These are illustrative operating gates, not statistical confidence thresholds or universal trading standards. The recorded code and data labels are frozen-v1 and synthetic-v1; in a real package replace labels with inspectable frozen artifacts and their actual content digests.
Serialize the candidate in a stable form and compute its digest. Save the readable fields beside it. Hashing makes an accidental configuration change detectable when compared with the saved reference; it does not stop someone with write permission from editing both files. Archive permissions and access control remain separate implementation questions.
Separate opening from successful completion
Create an opening receipt before executing outcome-dependent work. It contains the seal digest and a UTC opening timestamp. Then calculate the result and write a distinct result file. If execution stops between those operations, the receipt still shows that the period was opened, even though no valid performance decision exists.
The Python open documentation defines mode x as exclusive creation that fails when the path already exists. The sample uses it for both receipts and results. It never uses truncating mode w to replace a prior verdict. The file path is derived from the declared holdout identifier, which must be stable in a real archive.
with receipt_path.open('x', encoding='utf-8') as stream:
json.dump({'seal': seal_digest, 'openedAt': opened_at}, stream)This is a local exclusion mechanism, not a distributed transaction or tamper-proof service. Do not give the same physical reserved interval a new identifier to bypass it. The archive must bind identifiers to actual interval and data references; the small demonstration trusts its declared synthetic case identifiers.
Run the first synthetic decision
The first case, H1, supplies two invented net-profit observations, −1 and +0.5. Their total is −0.5. The evidence minimum of two is met, but the nonnegative-profit gate is not, so the result is fail. Save the count, amount, status and seal reference rather than only a displayed red badge.
The sample then tries to open H1 again with a more favorable set of observations. Exclusive creation rejects the attempt and leaves the original result unchanged. This test checks persistence of the first record, not whether the favorable observations represent a legitimate new dataset. Replacing an existing failure with them would be the prohibited behavior.
A second counterexample changes lookback from three to two while keeping the original seal. The digest check rejects it before the evaluation proceeds. If the researcher wants to develop that new lookback, create a new experiment with its own development history and independent future evidence. Reusing the already viewed interval can be exploration, but cannot be presented as a fresh independent confirmation.
Distinguish failure, insufficient information and interruption
Case H3 has one observation of +1. It returns information_insufficient, because a favorable amount does not satisfy the two-observation evidence requirement. Neither a pass nor a failure should be fabricated by ignoring the missing quantity of evidence. The same principle applies when quotes, corporate-action treatment or account-state inputs are not adequate.
Case H4 injects an execution exception after the opening receipt is created. Its receipt remains on disk and no result file exists. A second automatic attempt to open H4 is rejected. Report opened_without_result and investigate the execution fault; this is not evidence that the strategy lost money.
If a controlled technical replay is allowed, document it as a replay of an already opened test, preserve the original failure artifacts and keep selection frozen. A replay policy should be set before opening, including which corrections are purely technical. Our example deliberately blocks silent retries; it does not implement a general recovery manager or authorize a rerun after changing strategy logic.
Preserve the decision package for review
The package should contain the readable seal, configuration digest, opening time, original inputs, execution log, first output and a decision explanation. A reviewer should reconstruct why H1 failed and why H3 lacked information without asking the author to remember the rule. Preserve errors and unfinished receipts as carefully as successful results.
Keep selection and evaluation roles distinguishable even when one person performs both. Record when each action occurred and whether any manual intervention changed the candidate. If a rule was revised after seeing the holdout, disclose the revision and mark that result exploratory. A polished summary cannot undo the information exposure.
The sample creates its files inside a dedicated temporary directory, verifies their behavior and removes that demonstration directory at exit. A real research archive must retain its records. Do not copy the temporary cleanup policy into the persistent experiment store, and do not mistake the local test for a backup or durability audit.
Execute the checks and define the next step
Download the complete example. With Python 3.12 or later, run python R610.py. It uses only the standard library, synthetic observations and its own temporary files. Assertions verify H1’s −0.5 failure, rejection of repeat opening and modified seals, H3’s insufficient-information status, and H4’s retained opening without a result.
Accept the workflow when the first result survives attempted replacement, opening time and frozen inputs are traceable, and every attempted test has a distinct terminal or unresolved state. Do not interpret the two-observation toy gate as meaningful statistical assurance. The purpose is to verify process mechanics with inspectable outcomes.
After a failure, decide whether to retire the candidate or return to development under a new experiment. After insufficient information, follow the predefined evidence-collection rule without retuning against partial outcomes. After a pass, retain uncertainty and any further deployment requirements. Update the package when technical corrections are necessary, preserving the original opening and verdict rather than rewriting the history of what was learned.
FAQ
Does a hash prove that the holdout was never seen?
No. It identifies saved content. The information history and access process determine whether the interval was genuinely reserved.
Can an execution error be called a strategy failure?
No. An opened test without a valid result needs an explicit unfinished status and a documented technical investigation.