Learn / AI in Trading

How to evaluate an AI trading claim

Most of the distance between a published AI trading result and what you would experience is explained by six things. None of them require you to evaluate the strategy itself.

Apply them to anything, including this site.

1. Historical or forward?

The single most important question.

A language model has been trained on text written after the historical period being tested, including commentary explaining what happened. It may be recalling outcomes rather than analysing conditions, and nothing in the output distinguishes the two. You cannot instruct a model to forget.

So a backtest of a model on historical data is close to meaningless, regardless of methodology — see why AI backtests do not survive live trading.

Ask: was this evaluated on data after the model’s training cutoff? If the answer is unclear, treat the result as unevaluated.

2. Net of what?

“Net” usually means net of fees only. Three costs sit between gross and net:

A strategy with a genuine gross edge can lose money net. Nothing about the signal has to be wrong.

Ask: which of the three, and at what assumed rates?

3. How many runs, and were all reported?

Model output is a distribution, not a function — non-determinism and what it does to testing.

A single run is one sample. Several runs with the best reported is a sample selected for being good, and from the outside these look identical.

Ask: how many runs, and what was the worst one? The worst case matters more than the median, because it is what an account has to survive.

4. Over how many trades?

Twenty trades is consistent with almost any underlying edge. High-R, low-win-rate approaches need far more, because their return concentrates in a minority of trades — see how many trades before you can judge a strategy.

Ask: how many trades, and does the count suit the strategy’s shape?

5. What was the worst stretch, and what did recovery require?

Headline returns are reported; drawdowns are mentioned; the recovery a drawdown implies is almost never stated.

A 60% maximum drawdown means the account had to make 150% to get back — run it through the drawdown recovery calculator. Whether it plausibly could is a separate question from whether the backtest says it did.

Ask: what was the maximum drawdown, and would you have kept executing through it?

6. What is being sold?

Not a cynical question — an informational one. The claim’s shape follows the business model.

  • Selling signals or a bot — the claim is about returns, which is the hardest thing to verify and the easiest to select.
  • Selling software — the claim should be about capability, which is checkable.
  • Selling a course — the result is the marketing.

Ask: does the claim happen to be the least verifiable thing they could have said?

Red flags

A specific accuracy percentage with no sample size, period, or definition of “accurate”.

“AI-powered” as the mechanism. It describes the implementation, not the edge. Every claim above still needs answering.

Returns without drawdowns. Half a description.

Backtest-only results, for the reason in question 1.

No description of what it cannot do. A project that has thought carefully about its own limits usually says so — which is why Alpaca’s own MCP documentation warning that its server “can place real trades and access your portfolio” reads as a good sign rather than a bad one.

Guaranteed anything. Nothing in trading is guaranteed, and the word is a complete answer.

Applying it here

The claims on this site are deliberately of a checkable kind:

  • Exchange APIs behave in specific ways — verifiable against their docs, and where we could not verify something directly we say so, as with the IBKR connector details.
  • The arithmetic works out as shown — derivable, and we recompute it.
  • Limits enforced in code behave differently from limits in a prompt — an argument from how the components work.

We publish no price predictions, no signals, and no strategy performance. Not out of caution for its own sake: nobody has demonstrated a reliable way for a language model to forecast prices, and claiming otherwise would be selling something we cannot support.

If a claim here fails any of the six questions, it should not be there.

FAQ

How do I know if an AI trading tool actually works?

Ask what “works” means for it. Capability claims — does it report exposure correctly, catch costs, produce plans with defensible stops — are checkable in an afternoon. Return claims are the hardest to verify and the easiest to select favourably, so weight them accordingly.

Why are AI trading backtests unreliable?

Because the model was trained on text written after the period being tested, including explanations of what happened, so it may be recalling rather than analysing. That leak cannot be removed by careful methodology and cannot be instructed away. Forward testing after the training cutoff is the closest honest substitute.

What does “net of fees” usually leave out?

Funding and slippage. Funding accrues on perpetual positions held across settlements and can exceed trading fees; slippage is unquoted and concentrates on stop-outs, so it damages the losing side specifically. A result net of fees only can still be negative once both are included.

Is a high win rate a good sign?

It is uninformative on its own, and easy to manufacture by cutting winners early. What matters is expectancy — win rate combined with average win and average loss — which is why a strategy winning 80% of its trades can lose money and one winning 30% can make it.