Compare stock backtests only after checking their output files
A stock backtest marked successful can still lack the evidence needed for comparison. A summary number cannot replace missing trades, and a trade file from a different run cannot repair the gap. Validate the required artifact set against one run identity and coverage contract before admitting its results to a ranking.
This guide checks three synthetic artifact byte streams: summary, equity and trades. It detects missing output, truncated JSON and mixed-run substitution, then demonstrates recovery from a trusted same-run backup. This is an external method, not a verified CoTrading report-center capability or a real market performance audit.
Define the artifact contract before inspecting profit
List the required files and their purpose. The equity sequence supports return and drawdown calculations, the trade ledger supports execution and cost review, and the summary provides counts and selected metrics. Decide which checks each artifact must satisfy; a large summary profit should not make its missing supporting file optional.
Attach a stable run ID, data/configuration references, start and end coverage and expected schemas to the bundle. Store expected row counts and terminal values where appropriate. Preserve the manifest with the trusted run record rather than reconstructing it from whatever files happen to remain after a failure.
Distinguish an intentionally empty trade ledger from a missing one. A verified no-trade run can produce a valid file with zero rows and an explicit completion state. Absence of the file provides no such evidence. Do not infer that missing trades mean the strategy stayed flat.
Check file identity before comparing metrics
For every required artifact, verify existence, readable bytes and association with the same run and period. A matching filename is insufficient: exports from another attempt may use the same conventional name. Check the embedded identifiers and the references recorded by the producing task.
The Python JSON documentation provides parsing facilities, but successful parsing establishes syntax rather than research semantics. A well-formed object with the wrong run ID is still the wrong evidence. Validate schema, coverage and cross-file relationships separately.
Preserve original files before recovery. If an export is suspicious, avoid opening and resaving it through software that may change formatting or values. Record its observed digest and error, then investigate the producing attempt. Recovery should add traceable evidence rather than silently replacing the history of the failure.
Work through a complete same-run bundle
Our synthetic run is S1, covering indices zero through two. Its summary declares two trade rows, three equity observations and terminal value 102. Equity rows are (0,100), (1,101), (2,102). The tiny trade fixture records a purchase amount 100 and sale amount 102, consistent with a two-unit gain from opening value 100.
The demonstration uses JSON byte dictionaries so every injected defect is reproducible without touching real report directories. Its trade fixture is intentionally minimal; a production ledger also needs securities, timestamps, quantities, sides and fees. Do not describe this reduced schema as a complete real-trading export validator.
| Artifact | Embedded run and period | Expected content |
|---|---|---|
| summary.json | S1, 0–2 | counts 2 and 3, terminal 102 |
| equity.json | S1, 0–2 | three ordered observations |
| trades.json | S1, 0–2 | two fixture trade rows |
The checker admits the intact bundle. It verifies metadata, equity coverage, row-count agreement and terminal agreement. The fixture’s amount bridge is asserted independently. Matching these fields is a minimum evidence check, not proof that a real strategy’s accounting or prices are correct.
Use digests to detect unexpected replacement
The manifest retains SHA-256 digests of the trusted original bytes, using the interface documented in Python hashlib. Verify observed bytes against those values. A digest identifies content under the chosen serialization; it does not certify who produced it or whether the manifest itself is trustworthy.
if name not in files:
issues.append(name + ':missing')
elif sha256(files[name]).hexdigest() != expected[name]:
issues.append(name + ':digest_mismatch')Formatting-only reexports can also change a byte digest. If normalized semantic comparison is allowed, specify and validate that policy before using it. Do not automatically call every digest mismatch corruption, but do block admission until the replacement is explained. Never recompute the expected digest from an unexplained file merely to make the check pass.
Exercise three distinct failure cases
First remove trades.json. The bundle reports a missing required artifact and is blocked. Summary terminal 102 does not reconstruct the absent executions. The valid equity sequence also cannot reveal every fill, fee or intraperiod order, so it cannot substitute for the ledger.
Second replace the trade bytes with the unfinished text {"run_id":. Parsing fails and the changed digest is visible. A file can exist and still be truncated. File size alone is insufficient, and repairing the closing brace would not recover unknown missing rows.
Third provide well-formed trades labeled S0 instead of S1. The checker records a mixed-run issue. These rows might be perfectly valid for S0, but they do not belong to S1. Copying them into the right folder is not recovery; it creates a composite result that never existed as one execution.
Decide between restoration and rerunning
A trusted same-run backup can restore the exact missing bytes. The example retains the original S1 trade stream, reinstates it and verifies the expected digest and all cross-file checks. It does not generate trades from the summary or borrow another run’s output.
If an immutable checkpoint contains sufficient same-run execution state, a controlled reexport may be possible. Record the source checkpoint, exporter version, regenerated artifact and comparison policy. If those prerequisites are absent, rerun the frozen experiment rather than fabricating detail. Keep the earlier missing-artifact status in the audit history.
Distinguish a byte restoration from a new execution. A rerun may reproduce the same result under frozen conditions, but it has a new attempt identity. Label that relationship and retain the original failure. If data or engine changed, it is also a changed context that requires explicit comparison.
Build the output-admission record
Export required filename, existence, observed digest, parse status, run identity, coverage, row count and decision. Attach every issue and its recovery source. A reviewer should see why the initial bundle was blocked and what new evidence allowed admission.
Keep process status alongside artifact status. “Worker exited successfully” and “comparison evidence complete” answer different questions. If only summary-level comparison is intentionally allowed, narrow the claim and identify unsupported execution questions; do not quietly advertise a fully reproducible result without trades.
The admission record should name the exact comparison questions that remain supported.
Do not turn the checker into a license to alter original evidence. A correct recovery leaves enough trace to reconstruct both the damaged and restored states. Preserve access restrictions and avoid embedding credentials in manifests or shared diagnostics.
Run and accept the checks
Download the complete example. Use Python 3.12 or later and run python R619.py. Standard-library assertions admit the intact S1 bundle, block missing/truncated/mixed trade streams, restore the exact same-run backup and verify terminal 102 with two trade rows.
Accept only after the required artifacts belong to one run and period, their integrity policy passes and all cross-file counts reconcile. Archive the manifest and recovery explanation with the result. The sample manipulates synthetic byte streams; it does not inspect a live engine’s report service or certify actual historical performance.
Recheck when outputs are moved, regenerated, restored or exported under another format. Keep earlier integrity decisions rather than rewriting a successful-looking final folder. The comparison should remain explainable even when the original process reported success before its supporting files were safely preserved.
FAQ
Can a valid summary replace missing trade detail?
No. It can support only a narrower summary claim and cannot reconstruct execution evidence.
Is another run’s valid trade file an acceptable repair?
No. Restore verified same-run evidence or perform a clearly labeled rerun.