Why AI backtests do not survive live trading
Backtesting a language model on historical data is close to meaningless, because the model has already read the future. It was trained on text written after those events, including commentary explaining what happened and why.
That is a leak no careful methodology removes, and it is specific to models trained on broad text — a rule-based backtest has different and more tractable problems.
The leak that cannot be closed
A conventional backtest can be made honest with discipline: use only data available at each decision point, account for costs, avoid fitting to the sample.
None of that helps here. If you ask a model about a chart from a past period, it may recognise the period. Even if it does not recognise the specific instrument, it has absorbed the narrative — which regimes preceded which, which patterns were followed by which outcomes, what the consensus explanation became.
You cannot ask a model to forget. Instructing it not to use knowledge of the outcome does not remove the knowledge; it adds an instruction that competes with it. So a model that appears skilful on historical data may be recalling rather than analysing, and there is no way to tell which from the output.
This is the difference from a rule-based system. A moving-average crossover has no memory of 2022. A language model does.
The leaks that apply to any backtest, and are worse here
Survivorship. Instruments that were delisted or went to zero are missing from most historical datasets. A strategy tested on what still exists is tested on the survivors.
Costs left out. Fees on notional both sides, funding on perpetuals held across settlements, and slippage that concentrates on stop-outs. See the fee and funding calculators and slippage and market impact. Together these routinely convert a positive gross result into a negative net one.
Fill assumptions. Backtests fill at the price you asked for. Real stops gap; real market orders walk the book.
Selection after the fact. Trying several prompts and reporting the one that worked is fitting to the sample, and it is much easier to do with prompts than with parameters because it does not feel like optimisation.
Sample size. As covered in expectancy vs win rate, a few dozen trades is noise — and more so for high-R, low-win-rate approaches whose returns concentrate in a minority of trades.
The non-determinism problem
A rule-based backtest is reproducible: same data, same code, same result. That property is what makes it a test.
A model’s output is a distribution. Run the same evaluation twice and get different trades. Which means:
- A single backtest run is one sample, not a result.
- A good run may be the favourable tail.
- Regression testing is not available in the usual sense — you can only re-sample.
So an evaluation needs to be run many times and reported as a distribution. Most published results are a single run, and there is no way to tell from the outside whether it was the first one.
What can be evaluated honestly
The prediction framing is the part that does not survive. Other things do.
Forward testing on data after the training cutoff. Imperfect — the cutoff is approximate and later models absorb later data — but directionally sound and much better than historical evaluation.
Paper trading in real time. Slow, and the only way to observe the model responding to genuinely unseen conditions. See practising on testnet.
Non-predictive capabilities. Does it summarise exposure correctly? Catch costs you missed? Produce plans whose stops sit at defensible levels? These are checkable against ground truth and they are where the value actually is — see can AI predict the market.
Your own approval record. If a model drafts and you approve, the interesting question is whether the trades you rejected were systematically worse than the ones you approved. That measures the combined system and requires only a decision log — see logging tool calls for a trading audit trail.
Reading someone else’s results
Three questions that account for most of the gap between a claim and a result:
- Is this historical or forward? Historical evaluation of a language model is grading its own homework.
- Net of what? Fees, funding and slippage, specifically. “Net” without detail usually means net of fees only.
- How many runs, and were all of them reported? A single run is a sample.
A fourth, if you can get it: what happened in the worst stretch, and what recovery would that have required — the drawdown recovery calculator turns a headline maximum drawdown into the number that actually matters.
FAQ
Can I backtest an AI trading strategy?
Not in the way you can backtest a rule-based one. A language model has been trained on text written after the historical period you are testing, including explanations of what happened, so it may be recalling outcomes rather than analysing conditions — and nothing in the output distinguishes the two. Forward testing after the training cutoff is the closest honest substitute.
Why do AI strategies perform worse live than in testing?
Usually some combination of the model having prior knowledge of the test period, costs omitted from the simulation, optimistic fill assumptions, and a result reported from one favourable run of a non-deterministic process. Each is individually enough to account for a large gap; together they routinely turn a positive backtest into a negative live result.
How do I evaluate an AI trading tool honestly?
Test forward rather than backward, on data after the model’s training cutoff. Run the evaluation many times and look at the distribution rather than the best run. Include fees, funding and slippage. And evaluate the things that are checkable — whether it reports your exposure correctly, whether its plans have defensible stops — rather than only whether it was directionally right.
Is forward testing enough?
It is the best available option and it is not conclusive. Training cutoffs are approximate, models are updated, and a few months of forward results is still a small sample. It is meaningfully better than historical evaluation because the specific leak is absent, which is a lower bar than being reliable.