Compare effective settings between stock backtest runs
Two stock backtest configuration files can look different while producing identical effective settings. One may omit a default that the other states explicitly. Conversely, similar files can resolve differently because defaults, overrides or strategy-side values changed. Compare the values actually consumed, preserve their origins and use a controlled rerun before attributing performance to one setting.
This guide produces an effective-setting difference table, an allowed-change list and a synthetic confirmation run. It is an external research procedure, not a verified CoTrading export feature. Use one stock market and currency; the invented transaction below demonstrates configuration semantics rather than historical returns.
Identify every configuration source
Collect the raw files, command-line values, application defaults and strategy constructor arguments for both runs. Record the order in which overrides apply. A displayed configuration is insufficient if a strategy hard-codes a quantity or an adapter silently substitutes another fee. Freeze code, data, engine and execution policy alongside the settings.
Separate run-affecting fields from descriptive metadata. A report title can differ without changing the experiment, while a cost unit can change execution even if its numeric value is unchanged. Define that distinction in the schema, not by deleting fields that happen to make the comparison inconvenient.
The Python argparse documentation on defaults illustrates how an absent option receives a defined value. Our example implements its own dictionary resolver; it does not assume that every real engine uses argparse or that all strategy behavior is controlled by one configuration file.
Resolve defaults while retaining provenance
The synthetic defaults are lookback three, fee two basis points and quantity one. Resolving an empty object gives those three values, each labeled default. Resolving an object containing only fee_bps: 2 gives identical effective values, but the fee’s origin is explicit. Provenance differs; execution settings do not.
Keep both the normalized value and its source. Otherwise a later default change could make an old omitted field impossible to interpret. Save the version of the default set used for resolution. Normalizing today’s file with yesterday’s undocumented defaults cannot reconstruct what the older process consumed.
Treat null, false, zero and omission according to the field contract. They are not universally equivalent. Our positive integer lookback and quantity reject booleans, while fee zero would be a valid no-fee assumption. Do not use a generic “false-like means absent” rule that silently turns a deliberate zero charge into the default two basis points.
Reject unknown or ineffective fields
A misspelled fee_bp is rejected rather than ignored. The JSON Schema object reference explains restricting additional properties. We implement the rejection with a standard-library set comparison, without running a schema package.
unknown = set(raw) - set(defaults)
if unknown:
raise ValueError('unknown settings')
effective = {**defaults, **raw}An accepted field also needs an actual consumer. Trace fee values into the execution calculation and quantity into the order amount. A field present in JSON but never read by the strategy is not an effective setting. Record it as unsupported or ineffective, then correct the adapter rather than presenting a misleading parameter comparison.
If units require conversion, retain both representations and the conversion rule. Two fee fields expressed as percent and basis points cannot be compared as bare numbers. This sample keeps both sides in basis points so the only intended change remains unambiguous.
Build the comparison table
| Case | Raw fee | Effective fee | Effective quantity | Interpretation |
|---|---|---|---|---|
| A | omitted | 2 bp | 1 | default-resolved baseline |
| B | explicitly 2 | 2 bp | 1 | no effective difference |
| C | explicitly 4 | 4 bp | 2 | two settings changed |
| D | explicitly 4 | 4 bp | 1 | permitted fee-only change |
The lookback remains three in this four-case comparison. A versus B has an empty effective difference, even though the raw documents differ. A versus C changes both fee and quantity, so a claim that the fee alone explains a profit change is unsupported. The allowed-change set contains only fee_bps, and the checker rejects C.
Case D is constructed from the baseline with just that field changed. Preserve C as the original uncontrolled comparison rather than rewriting its configuration to pretend it was controlled. D is a new confirmation experiment with its own reference and a clearly stated intervention.
Run the one-field confirmation
The known signal history is [90,95,100]; with lookback three, the last value exceeds its window mean and permits the fixture’s round trip. The synthetic execution buys one share at 100 and sells at 110. Gross profit is ten. Both sides contribute to traded notional, totaling 210. At two basis points, cost is 0.042 and net profit 9.958; at four, cost is 0.084 and net profit 9.916.
The net difference is −0.042, exactly the additional charge on the same frozen transaction. The example returns an effective-configuration receipt from the execution function and asserts that it matches the resolved request. A receipt copied before an adapter discards arguments would not establish that relationship.
A separate field-consumption check uses lookback one, produces no signal and returns net zero. It verifies that the accepted lookback affects calculation; it is not part of the fee-only comparison.
This is a fixed-fill calculation. A real fee change could alter affordability, sizing or later orders. When that happens, rerun the full engine under the controlled configuration and explain the resulting path. Do not infer that every fee-only experiment must differ by a simple final subtraction.
Inspect multiple differences before assigning a cause
The different-mechanism counterexample is C’s quantity two. It changes gross exposure as well as fees. Even if its final profit were close to the baseline, closeness would not restore the one-field premise. Check the allowed-change set before examining whether performance moved in the expected direction.
If several changes are intentional, name the intervention accordingly or run a series of controlled experiments with explicit dependencies. Do not silently select the most plausible field from a large difference table and call it the cause. A setting comparison can identify candidates for explanation; causal attribution depends on the experiment and its held conditions.
If effective values match but outputs differ, stop blaming the raw configuration formatting. Investigate consumed data, code, runtime, randomness and execution artifacts. Conversely, an empty output difference does not prove the settings were equal; distinct configurations may produce the same trades on a small price path.
Export a reviewable setting ledger
For every field, save raw presence, raw value, resolved value, origin, unit, schema/default version and consuming component. Mark whether the difference is permitted, descriptive or blocking. Attach both original run references and the controlled confirmation run.
Keep sensitive values out of shared exports. A reusable research ledger can reference a data-access prerequisite without embedding credentials. Do not replace missing consumed values with guessed defaults; an unresolved historical setting is a limitation that may prevent attributing an older result.
Validate the table with an omitted/default pair, an actual permitted difference, an unauthorized second change and an unknown key. These cases distinguish normalization, control and input validation rather than relying on one happy-path comparison.
Execute and accept the result
The fee parser first converts the supplied value to Decimal, then requires a finite, nonnegative result. Numeric and textual NaN and positive or negative infinity are rejected with a clear ValueError before comparison, migration or execution; a sign check alone is insufficient for NaN. The regression fixture exercises all six forms on each applicable parsing path.
Download the complete example. Use Python 3.12 or later and run python R617.py. It uses only standard-library decimal arithmetic and synthetic inputs. Assertions verify the empty omitted/default difference, rejection of quantity changes and unknown keys, matching effective receipts, and exact net amounts 9.958 and 9.916.
Accept the package when the compared settings are resolved under recorded rules, all effective differences are explained, the confirmation changes only the allowed field and its outputs reconcile. Preserve the provenance distinction even when it does not count as a behavioral difference.
Rebuild the ledger when defaults, schema, override order, strategy code or execution behavior changes. Do not normalize every historical run with the newest rules and overwrite its original interpretation. This method verifies a setting-to-calculation path on invented data, not real market profitability or a completed product integration.
FAQ
Does an omitted default count as a parameter change?
Not when both runs resolve to the same value under their recorded rules. Its provenance should still be preserved.
Can one field explain a result when another also changed?
Not from that comparison alone. Reject the one-field claim or run a controlled confirmation.