Audit failed trials before ranking a stock backtest batch
A batch’s largest available profit is not necessarily its best result when half the planned trials failed. Reconcile every planned configuration to a known state, decide which outputs are eligible for comparison and disclose the remaining gaps. A failed or missing run has no measured return; assigning it zero can alter the ranking while concealing the reason evidence is absent.
This guide builds a batch-state ledger, failure-routing table and minimal technical rerun queue from synthetic records. It is an external research method, not a verified CoTrading batch-audit feature. Keep market, currency and experimental context fixed, and distinguish operational completion from evidence that a stock strategy is profitable.
Start from the frozen plan
Obtain the list of intended trial identifiers before reading result files. Each identifier should connect to a legal configuration, code, data, execution policy and requested interval. Result directories alone cannot reconstruct missing planned trials, because the very failures under investigation may have created no directory.
Keep candidate identity separate from attempt identity. A technical retry of one configuration can create two attempt records without adding another candidate to the search. Preserve both histories, then apply the documented rule for the candidate’s final state. Do not count the successful retry and its earlier failure as two independent parameter locations.
Record dispatch acknowledgments or an authoritative scheduler registry. No log is not proof that a task never started. A process could have started and failed before emitting its first file. If dispatch state is unknown, keep it unresolved and block a final completeness claim rather than relabeling it as not started.
Define the state model and admission rule
For this completed audit, each planned trial must finish in one of four groups: admitted completion, failure, cancellation or confirmed not started. A still-running or unknown state means the final audit is not ready. A progress report may list such states separately; it must not force them into a terminal category to make an equation balance.
Process success and artifact success are different. The Python subprocess documentation describes return codes and captured outputs. A zero process code alone does not certify that equity, trades and required summaries are complete. Admission therefore checks the artifacts and a finite comparable score after execution is reported complete.
Freeze the artifact contract and ranking metric beforehand. A missing trade ledger must not suddenly become optional because its summary profit is large. Conversely, a completed no-trade strategy can have a valid zero result when its full artifacts and status support that interpretation.
Work through eight planned trials
The plan contains T1 through T8. T1 completes with valid outputs and net score five. T2 completes with valid outputs and score negative two. T3 fails during input loading, T4 during strategy execution and T5 during the engine stage. T6 reports execution complete but fails its artifact check, despite a summary value of nine.
T7 is cancelled and T8 is explicitly confirmed not started. The final evidence ledger is:
| Final group | Trials | Count |
|---|---|---|
| Admitted completion | T1, T2 | 2 |
| Failure | T3, T4, T5, T6 | 4 |
| Cancelled | T7 | 1 |
| Confirmed not started | T8 | 1 |
Eight equals two plus four plus one plus one. Admission completion is 25%, not the fraction of processes that happened to report success. T6’s impressive summary does not enter ranking because the required evidence is incomplete.
Route failures by the first observable stage
Keep loading, strategy, engine and artifact failures separate. A loading failure suggests input acquisition or parsing investigation; a strategy exception suggests a code contract problem; an engine failure needs execution diagnostics; an artifact failure concerns output construction or preservation. These are routing labels based on available evidence, not claims of an ultimate root cause.
The synthetic policy marks T3’s loading failure and T6’s artifact failure technically retryable. T4 and T5 require investigation before another automatic attempt. The minimal queue is therefore [T3,T6]. This is a declared illustrative policy, not a universal rule that all loading failures are safe to retry.
For T6, determine whether trusted checkpoint state can regenerate the missing output or whether the full execution must be rerun. Never fill its folder with another trial’s trades. For T3, a data revision or changed permission scope creates a new context that must be recorded; it is not automatically the same frozen experiment.
Implement reconciliation before ranking
The first guard compares the plan and registry identities exactly:
if len(set(plan)) != len(plan) or set(records) != set(plan):
raise ValueError('duplicate, orphan or unresolved trial')Then classify each record, apply artifact admission and sum the final categories. Preserve original process status alongside final evidence status. A report can then explain why T6 changed category without erasing that its worker originally claimed completion.
Reject orphan outputs whose IDs are outside the frozen plan. Investigate whether they belong to an earlier batch, an accidental duplicate or an undocumented expansion. Do not quietly import their favorable scores into the candidate table, because that changes both the experiment family and the completeness denominator.
State what the available ranking can support
Among admitted T1 and T2, T1 has the larger measured score. The honest label is best completed trial under the current admission contract. It is not best of all eight planned configurations: the unmeasured candidates could have scored differently, and cancellation or non-starting may be related to parameter complexity.
Do not average missing returns as zero, remove failed rows without disclosure or compare completion-only averages as though failure were random. Report attempted and admitted coverage by parameter region where useful. If every high-lookback candidate fails, the surviving sample may systematically favor short lookbacks rather than fairly evaluate the intended domain.
A final claim of exhaustive batch ranking requires resolving the missing candidates under the frozen or explicitly revised protocol. Until then, a provisional candidate can be named for further work, with the coverage limitation next to it. Operational admission does not remove multiple-search bias or provide out-of-sample evidence.
Test the missing-registry counterexample
The different-mechanism counterexample removes T8’s registry record. The plan still contains eight IDs, but the audit now rejects the unresolved identity. It does not infer not started merely because T8 has no execution output. This protects against a quiet disappearance before logging.
The sample also retains T6’s summary score nine outside the eligible list. A comparison that selected nine would fail the expected admission assertion. Together these checks cover identity completeness and artifact validity, which are distinct from the strategy-stage exceptions already listed.
In a production audit, preserve the original error text, timestamp, attempt number and artifact references. Sanitize credentials before sharing logs. Avoid rewriting an old failure record when a rerun succeeds; link the new attempt and update the candidate’s current status with an explainable transition.
Execute and accept the audit
Download the complete example. Use Python 3.12 or later and run python R618.py. It uses standard-library collections and synthetic registry records. Assertions verify eight-trial reconciliation, eligible T1/T2, rerun T3/T6, T1’s limited ranking and rejection of a missing registry entry.
Accept the package when every planned candidate has an evidenced state, all admitted metrics have complete artifacts, reruns are justified and ranking language matches coverage. Export the plan, attempts, final states, failure routes and original errors together rather than only a winner table.
Reaudit when retries finish, cancellations are confirmed, artifacts are recovered or the plan expands. Preserve each earlier audit version. This procedure verifies synthetic state accounting; it does not diagnose real data permissions, market rules or product services that have not actually been inspected.
FAQ
Can missing trials be assigned zero return?
No. Zero is a measured result under a valid run, not a substitute for absent evidence.
Does no log prove that a trial never started?
No. Confirm dispatch state from an authoritative record or retain an unresolved status.