Non-determinism and what it does to testing
Run the same evaluation twice against a language model and you get different trades. That single property removes most of the testing apparatus that makes rule-based systems trustworthy, and it is rarely accounted for in published results.
What breaks
A rule-based system is a function: same inputs, same output. That property is what makes a backtest a test rather than an anecdote, and what makes a regression suite meaningful.
A model’s output is a distribution over outputs. Which means:
One evaluation run is one sample. A good result may be the favourable tail rather than the expected value.
Regression testing is unavailable in the usual sense. You cannot verify a change did not break something by re-running and comparing — the output differs anyway. You can only re-sample and compare distributions.
Bugs may not reproduce. A wrong parameter that appeared once may not appear again, so you cannot write a failing test, fix it, and watch it pass. It is easy to conclude it was a fluke.
“It works” is ambiguous. Works how often? At what variance? Those are the questions, and a demo does not answer them.
Temperature is not the fix
Setting temperature to zero reduces variability without eliminating it, and does not restore determinism in practice — batching, hardware differences, model updates and floating-point ordering all contribute.
Even if it did, you would have traded one problem for another: a deterministic model at temperature zero is still not verifiable, because you cannot read the rule it is following. Determinism is necessary for testability, not sufficient.
More usefully: low variance in a trading context is not automatically desirable. It means the same input reliably produces the same output, not that the output is correct. A confidently consistent wrong answer is worse than an inconsistent one, because consistency reads as reliability.
What this does to evaluation
Any claim about model trading performance should be read with three questions, and the third is the one usually missing:
- Was this historical or forward? Historical evaluation of a model is grading its own homework — see why AI backtests do not survive live trading.
- Net of fees, funding and slippage?
- How many runs, and were all of them reported?
A single run is a sample. Ten runs with the best one reported is a sample selected for being good. Neither is a result, and there is no way to tell from the outside which you are looking at.
The honest form is a distribution: median, spread, worst case. The worst case matters more than the median, because it is what your account has to survive — which is the same argument as risk of ruin applied to model behaviour.
What it means for your own setup
Do not tune prompts on small samples. Trying variations and keeping the one that worked is fitting to noise, and it is easier to do accidentally with prompts than with parameters because it does not feel like optimisation. See how many trades before you can judge a strategy — the same arithmetic applies, with an extra source of variance stacked on top.
Evaluate the checkable parts. Does it report exposure correctly? Catch the funding you missed? Produce plans whose stops sit at defensible levels? These have ground truth and can be verified per run.
Do not rely on consistency you have not measured. If your process assumes the model will always include a stop, verify how often it does across many runs. “It always has so far” over a dozen observations is not a guarantee — and this is exactly why a schema enforces the field instead of a prompt asking for it.
The design conclusion
Non-determinism is not a defect to engineer around. It is a property of the tool, and it determines where the tool belongs.
Put the model where variance is acceptable — reading, summarising, explaining, drafting. In those roles a different-but-reasonable output each time is fine.
Put code where variance is not acceptable — risk limits, position sizing arithmetic, order construction, retry decisions. These must behave identically every time, and only one kind of component does.
That division is the whole argument in MCP server vs trading bot, arriving from a different direction. And it is why a limit in a prompt is not a limit: a probabilistic process cannot provide a guarantee, however reliably it has behaved so far.
FAQ
Why does the AI give different answers to the same question?
Because generation samples from a distribution over possible outputs rather than computing a single answer. Lowering temperature narrows the distribution without eliminating it, and other sources — batching, hardware, model updates — contribute regardless. It is a property of the tool, not a misconfiguration.
Can I make an AI trading system deterministic?
Not the model itself, in any way you should rely on. What you can do is make the parts around it deterministic: schema-validated output, sizing computed in code, limits enforced as functions. The model’s variability then affects which trades are proposed rather than whether the rules hold.
How many test runs do I need to evaluate an AI trading setup?
Enough that the distribution stabilises rather than a fixed number, and substantially more than one. Report median and worst case rather than a single result — the worst case is what your account must survive, and a single run tells you nothing about it.
Does temperature zero fix this?
It reduces variability without producing reliable determinism, and it would not solve the underlying problem if it did. A consistent output is not a verifiable one: you still cannot read the rule being followed, and a confidently consistent wrong answer is harder to notice than an inconsistent one.