Grid or random search under a fixed stock backtest budget
With a small stock backtest budget, compare what each search route actually evaluates before comparing its largest score. A grid can cover a few axis values systematically, while random draws may reach different combinations or spend attempts on duplicates. Neither route has been fairly compared if one quietly receives replacement runs after failures and the other does not.
This guide builds two six-attempt queues on the same sixteen-point parameter domain. Its invented score function measures the bookkeeping exercise only; it is not historical stock performance. Run one market and currency at a time. Search orchestration is an external research method, and CoTrading support remains planned or unverified.
Write the experiment contract before choosing a sampler
Freeze the strategy source, dataset, execution engine, observation interval, costs and objective. Resolve the legal parameter domain once, then give the same domain to both routes. A route should not obtain an easier scoring period, a larger stock universe or a different minimum-trade filter simply because its configuration generator has a different interface.
The example uses fast lookbacks 2, 3, 4 and 5, with slow lookbacks 6, 8, 10 and 12. Every fast value is below every slow value, leaving sixteen legal pairs. This removes legality filtering as a confounding difference. If a real domain has invalid combinations, apply the same constraint before sampling and retain the rejection counts separately.
The scikit-learn parameter-search documentation distinguishes exhaustive grids from randomized candidate evaluation. We implement our own small samplers with Python’s standard library. No scikit-learn model or market engine is executed, and our grid is a deliberately coarse six-point design rather than an exhaustive search of all sixteen pairs.
Decide what one unit of budget buys
Our unit is one attempted execution, whether completed, failed or repeated. Both routes receive exactly six attempts. A failed point consumes one attempt; a repeated point also consumes one because we actually evaluate it again. This policy is simple to reconcile and intentionally makes duplicate spending visible.
Other policies are possible, such as six distinct completed configurations or a fixed wall-clock allowance. They answer different questions. A completed-run budget can hide expensive failures; a time budget depends on hardware and scheduling. Declare the policy before viewing results, record both attempted and completed counts, and avoid describing different policies as equal budgets.
If production retries are necessary, distinguish infrastructure retries from new candidate evaluations. Decide whether retries consume the budget and cap them. A route that loses one task to an outage should not silently receive unlimited replacement opportunities. Keep attempt IDs beneath candidate IDs so repeated executions cannot inflate the number of unique locations explored.
Construct the two queues explicitly
The grid takes fast values 2 and 5 crossed with slow values 6, 8 and 12. This gives six distinct pairs, reaches both ends of the fast domain and covers three slow values, but never visits fast 3 or 4 or slow 10. Its holes are design consequences rather than execution failures.
The random route uses Random(17).choices(legal, k=6), drawing uniformly with replacement from the same sixteen legal pairs. The recorded queue is (4,6), (5,6), (5,12), (3,6), (5,6), (4,12). The pair (5,6) occurs twice. Preserve that actual queue; a seed alone is insufficient documentation if the legal domain order or sampling implementation later changes.
legal = list(product([2, 3, 4, 5], [6, 8, 10, 12]))
grid = list(product([2, 5], [6, 8, 12]))
random_queue = Random(17).choices(legal, k=6)Sampling without replacement would remove duplicates, but that would be a different sampler. It can be sensible on a small finite domain; do not change to it after seeing an inconvenient repeated draw and continue reporting the original experiment name. Record whether probability is uniform over legal configurations or over independent axis draws followed by rejection.
Evaluate under one shared function
The synthetic score is 10 - 2*abs(fast-3) - abs(slow-10)/2. We also inject a deterministic failure at (5,12). Both queues contain that point, so both have one failed attempt. Failed scores remain missing, not zero. A zero could be a legitimate evaluated score and would contaminate ranking if used as an error substitute.
Keep a row for every attempt with candidate values, status, output and error reason. The example computes coverage from requested coordinates, then separately counts successful completion. In a real report, show attempted coverage and successfully measured coverage together. A parameter region reached only by failed jobs has not yielded performance evidence.
The function is intentionally cheap and deterministic. Consequently this exercise compares candidate placement and accounting, not compute efficiency under an expensive engine. A production study would need actual elapsed time, resource policy and failure causes before claiming that equal attempt counts also represent equal computational spending.
Reconcile the observed coverage
The actual grid result is six attempts, six unique pairs, zero duplicates and one failure, spanning two fast values and three slow values. The random result is six attempts, five unique pairs, one duplicate and one failure, spanning three fast values and two slow values. Both have five completed attempts, but the repeated successful point means the random route has fewer distinct completed observations.
| Route | Attempts | Unique proposed pairs | Repeated attempts | Failed attempts |
|---|---|---|---|---|
| Coarse grid | 6 | 6 | 0 | 1 |
| Random, seed 17 | 6 | 5 | 1 | 1 |
Axis counts alone do not describe joint coverage. Three fast values at two slow values are not interchangeable with two fast values at three slow values. Retain the pair list or a coordinate map with explicit unvisited and failed states. Do not smooth over unvisited combinations or infer their scores from nearby results.
Use the duplicate counterexample to test the ledger
The different failure mechanism is a queue containing the same legal point six times. It has six attempted runs, one unique configuration and five duplicate attempts. Its executions might all succeed, but calling it six-point coverage would be wrong. The sample asserts this count independently of the injected failure case.
Duplicates may be intentional when estimating stochastic execution variability. In that case they are replications with their own purpose, not additional parameter discoveries. Separate the replication budget from exploration and retain the random-state policy for each replicate. Our deterministic example receives no new information from repeating a completed point.
Also test an accidentally shortened queue and a queue outside the shared domain before dispatching real work. The comparison should stop if route lengths violate the agreed budget or if a sampler emits an inadmissible pair. Correct the construction problem first; do not repair it by dropping inconvenient outputs from the final table.
Interpret the comparison without overclaiming
The Bergstra and Bengio paper’s original abstract discusses random search in neural-network hyperparameter optimization. It motivates considering parameter relevance and coverage, but it does not establish that our stock strategy will earn more under random search. Only the abstract page was consulted for that contextual statement.
For this exercise, report the queues, counts, gaps and common evaluator. If a highest score is shown, label it exploratory and retain the other attempts. One seed’s maximum cannot establish a search algorithm’s lasting superiority. Expanding the study to several seeds adds experiments and must be budgeted and recorded, not treated as a free robustness check after the preferred result appears.
Accept the batch when both attempt totals equal six, every point belongs to the frozen domain, failures and duplicates reconcile, and the queue can be replayed. Rebuild the context if domain ordering, data, code, objective or retry policy changes. Candidate selection and an independent holdout remain later steps.
Download the complete example. Use Python 3.12 or later and run python R606.py. It uses only the standard library and synthetic inputs; its assertions verify queue accounting rather than market returns.
FAQ
Should random sampling always avoid replacement?
No. The policy depends on the purpose. Distinct exploration and deliberate stochastic replication should have separate accounting.
Do equal attempt counts mean equal compute cost?
Not necessarily. Runtime, retries and resources can differ. This example fixes attempted evaluations, not wall-clock consumption.