Why do identical stock strategy backtests rerun differently?
Two runs can use the same strategy filename and still execute different experiments. The data may have been revised, resolved configuration may differ, a dependency may have changed or a hidden execution setting may alter fills. Diagnose the first differing layer before interpreting a difference in final profit. A final metric compresses too much information to identify the cause by itself.
This procedure compares already frozen stock programs. Save generated code before executing it; asking an external model to generate code again is a different stage and does not satisfy the same-program premise. CoTrading behavior is unverified here. The standalone Python demonstration uses synthetic observations and a deliberately simple execution rule, with no market feed, model call or broker test.
Define what the two runs claim to share
Create one record for each run containing code bytes, validated configuration, data bytes, engine build, dependency versions, interpreter, calendar and execution policy. Include random-state ownership, worker count, relevant locale or timezone settings and initial account state. Capture references to the values consumed by the process, not merely the settings displayed in a user interface.
Keep secrets out of the reproduction. An account token is unnecessary for replaying an already captured synthetic or authorized offline dataset. If an input cannot be frozen because the program calls a live service or external model during execution, label that dependency as unfrozen. The diagnostic conclusion may be that deterministic replay is not yet a valid expectation, rather than that the engine is defective.
Choose comparison rules in advance. Dates, security identifiers, quantities, statuses and signal directions usually need exact comparison. Floating quantities require an explicit tolerance with units. The Python isclose documentation describes relative and absolute tolerances. Near zero, a relative tolerance alone can be unhelpful; choose an absolute limit justified by the calculation, not by the difference you want to dismiss.
Compare artifacts in causal order
Use this order: consumed data, resolved configuration, declared runtime, signals, fills and account equity. An earlier difference can explain several later ones. Finding changed fills does not establish a matching-engine cause if signal inputs already differ. The first difference is the earliest observed discrepancy under this instrumentation, not proof that no hidden upstream dependency exists.
| Layer | Compare | Useful first action |
|---|---|---|
| Data | Values, order, timestamps, adjustment version | Locate the first changed observation |
| Configuration | Resolved parameters, costs, starting state | Trace defaults and overrides |
| Runtime | Code, engine, dependencies, random policy | Recreate the recorded environment |
| Signals | Timestamp, security, direction, size | Compare the generating input window |
| Fills | Signal reference, time, price, quantity, fee | Replay execution with identical signals |
| Equity | Cash, holdings, marks and aggregation | Reconcile the first state transition |
Keep row counts and stable event keys as well as hashes. A digest shows whether serialized content differs but does not tell you which row changed. The hashlib documentation supplies the digest mechanism used in the example; the causal layer design and comparison policy are our own research choices.
Normalize representation without erasing meaning
For structured snapshots, serialize dictionaries with sorted keys and consistent separators before calculating a digest. Preserve array order whenever it represents event order. Reordering two same-time orders can change cash availability or fill priority, so sorting all rows to make hashes agree can destroy the evidence you need.
Retain both original bytes and normalized comparison artifacts. File formatting, key order and display timestamps can change bytes without changing the experiment. Conversely, rounding all prices to two decimals may hide a meaningful execution difference. Document which transformations are representational and why they preserve the strategy’s semantics.
Compare exact hashes first where suitable, then inspect row-level values. Apply numeric tolerances to the selected numeric fields rather than to an entire JSON object. A close final balance cannot excuse a missing order or a changed security identifier. Conversely, a harmless floating summation difference does not necessarily justify a claim that the strategy itself changed.
Reproduce a data-revision case
The example uses synthetic prices [100, 102, 101, 104], initial cash 1000 and seed 7. At indices one and two, a signal is one when the current price exceeds the previous price. Each positive signal purchases one share at the following observation plus a small seeded synthetic price increment. This rule preserves the distinction between signal time and a later fill time, although it omits real liquidity and market restrictions.
Run the baseline twice. Both runs produce signals [1, 0]; every layer and the normalized digest agree. Next revise the third price from 101 to 103, producing [100, 102, 103, 104]. The signal sequence becomes [1, 1]. The earliest difference is the data layer, even though both fills and final equity also change.
The appropriate fix is to identify why the consumed snapshot changed. Possible research actions are restoring the original data for replay or explicitly declaring a revised-data experiment. Changing the random seed or loosening the final-equity tolerance would not address this demonstrated cause. Preserve both snapshots and the observation-level difference so the revision remains explainable.
Exercise a different mechanism: hidden execution cost
The counterexample leaves data, declared configuration, runtime record and signals unchanged but injects a fee of 0.25 into the execution function. Only one baseline buy occurs. The first differing recorded layer is fills, and final equity is lower by exactly 0.25. The example intentionally leaves the fee out of the declared configuration to expose an instrumentation gap.
This does not prove a fee can legitimately remain hidden. A repair would move every execution setting into the resolved configuration, freeze it and rerun. Once that repair exists, a different fee should be reported as a configuration difference earlier in the chain. Until then, the fill-level discrepancy tells the investigator where to inspect the unrecorded behavior.
Use the same signals as a replay input to isolate matching and accounting. If fills match but equity differs, inspect cash movements, position updates and valuation marks next. Avoid modifying several layers simultaneously: that can make the final result agree while leaving the original cause unexplained.
Run the minimal diagnostic package
The core comparator is intentionally small:
LAYERS = ["data", "config", "runtime", "signals", "fills", "equity"]
def first_difference(left, right):
return next((layer for layer in LAYERS
if left[layer] != right[layer]), None)Run python docs/growth/research/round-5/examples/R622.py. The actual output is:
repeat_first_difference=None normalized_digest_equal=True
revised_first_difference=data signals=[1, 0]->[1, 1]
hidden_fee_first_difference=fills final_equity_delta=0.250000
reordered_keys_normalized_equal=TrueThe full script implements the simulation, snapshot digests, injected differences and assertions. It also constructs the same object with reversed dictionary key insertion order and verifies equal normalized digests. That representation check has a different interpretation from either changed prices or changed fees; it prevents a file-order artifact from being reported as changed trading behavior.
Control random state and close the incident
The demonstration creates a private Random(7) instance for each run. The Python random documentation discusses seed-based reproducibility and version limitations. A seed alone does not freeze thread scheduling, other libraries or external model responses. Record the runtime and ownership of each random stream; concurrent workers consuming a shared stream can assign draws to different events.
Accept the repair only after a fresh run under the corrected frozen context matches the baseline at all required layers, using the original comparison policy. Save the two manifests, smallest reproducing input, first changed event, downstream impact, explanation and repaired-run evidence. Mark invalidated conclusions rather than overwriting the old run.
Reopen the check when data, configuration, dependency, engine, concurrency or execution policy changes. If only normalized artifacts agree while raw bytes differ, explain that limited conclusion. If an external dependency remains unfrozen, state that the package reproduces only the captured offline path. Passing this synthetic package verifies its assertions; it does not establish real historical profitability or deterministic behavior of any untested product workflow.
Make the first difference independently inspectable
Attach the stable event key, both old and new field values, their immediate inputs and the comparison rule to the incident record. In the price-revision example, the changed observation is index two, 101 versus 103; the affected signal at that index changes from zero to one. In the fee example, the existing buy keeps its timing and quantity, but its fee becomes 0.25. A reviewer can inspect those local transitions before trusting the aggregate equity difference. Preserve an executable entry command and its working-directory requirement so the reproduction is more than a collection of screenshots.
Download the complete example. Use Python 3.12 or later and run python R622.py in the download directory. It uses only the standard library and synthetic inputs.
FAQ
Should I set a seed before investigating anything else?
Record random state, but compare the consumed data and resolved settings first. A seed cannot repair revised prices or ignored configuration.
Does the first differing layer prove the ultimate cause?
It identifies the earliest discrepancy visible in the saved layers. Hidden dependencies may require additional instrumentation before the ultimate cause is known.