Backfilling trade and price history
Figures on this page are as of 2026-09. Fees, limits and margin tiers change — check the venue's own docs before acting on a number.
Fetching history looks like a loop over pages. It is a loop over pages with three complications that produce silently incomplete data: cursor semantics, archive endpoints for older records, and retention windows shorter than you expect.
Silently incomplete is the operative phrase. A gap in historical data does not error.
Pagination, and the shape that goes wrong
Venues paginate by time range, by cursor, or by ID. The failure is the same in each: an off-by-one at the boundary silently drops or duplicates a record.
Time-range pagination. Are the bounds inclusive or exclusive? Getting this wrong at every page boundary produces a gap or a duplicate every page — and for trade history that means a wrong position history.
Cursor pagination. Cleaner, provided you carry the cursor exactly and do not reconstruct it.
ID pagination. Usually the most reliable, since IDs are monotonic within a symbol.
Three rules that prevent most of the damage:
- Page in one direction only. Mixing forward and backward paging across a live boundary is where duplicates come from.
- Deduplicate on the venue’s own ID, not on a composite you built.
- Verify continuity after collection. For time series, check that intervals are contiguous. A gap is a bug, not a quiet market.
Archive endpoints
The one people miss.
Several venues serve recent history from one endpoint and older history from a separate archive endpoint. Bybit documents archive variants for fills and order history; OKX and Binance have their own arrangements.
If you page backwards through the normal endpoint, you do not get an error when you cross the boundary. You get an empty result, which looks exactly like “no more data”.
So a backfill that stops after a few weeks may have hit the recency boundary rather than the beginning of your history — and the data looks complete.
Check whether your venue has an archive endpoint before concluding your history starts where the results stop.
Retention
History is not kept forever, and the limit is usually shorter than assumed. Some venues retain detailed fills for months rather than years.
The practical consequence: if you are not archiving your own data, you are dependent on the venue’s retention, and you will discover its limit at the moment you want a long backtest.
Start storing your own fills from day one. It costs nothing and it is unrecoverable later. This is the same argument as what to record in a trading journal, arriving from the infrastructure side.
Rate limits during backfill
A backfill is the most rate-limit-intensive thing a trading integration does, and it is usually run once, badly.
On Binance, metering is weight-based and
historical endpoints often carry higher weight per call. Exceeding it returns
-1003, and severe violations escalate to an HTTP 418 IP ban that outlasts
the burst.
Rules:
- Throttle deliberately. A backfill has no deadline; there is no reason to run it at full speed.
- Never backfill from the process that trades. A backfill that earns a rate limit takes your trading capacity with it.
- Make it resumable. Store progress so an interruption does not restart from zero and re-spend the budget.
Candles have their own problems
The current candle is incomplete and changes until its interval closes. Storing it as final produces a wrong bar that never corrects.
Timestamp convention varies — open time or close time, and venues differ. Getting it wrong shifts your entire series by one interval, which is the kind of error that survives a long time because everything still looks plausible.
Gaps are real. Missing candles occur during outages. Your code should distinguish “no data” from “zero volume”.
Verifying a backfill
Do not assume it worked:
- Count the records you expected against what you got, for a known period.
- Check continuity — no gaps in a time series, no missing IDs in a sequence.
- Spot-check against the UI for a specific day.
- Reconcile the endpoint: the position implied by summing your backfilled fills should match the current position. If it does not, the history is incomplete — the same logic as position reconciliation.
That last check is the strongest, because it validates the whole series against an independent number.
FAQ
Why is my trade history incomplete?
Most often because older records live on a separate archive endpoint, and paging past the recency boundary on the normal endpoint returns empty results rather than an error — which looks identical to reaching the start of your history. Retention limits produce the same symptom.
How do I paginate exchange history correctly?
Page in one direction only, deduplicate on the venue’s own record ID, and verify continuity after collection rather than trusting the loop. Confirm whether range bounds are inclusive or exclusive, since an off-by-one at every page boundary produces a gap or duplicate per page.
How far back does exchange trade history go?
Less far than most people assume, and it varies by venue and record type — some retain detailed fills for months rather than years. Since it is unrecoverable once aged out, store your own fills from the beginning rather than relying on the venue’s retention.
Will backfilling hit my rate limit?
Almost certainly if you run it at full speed, since historical endpoints often carry higher weight per call. Throttle deliberately, make the job resumable so an interruption does not re-spend the budget, and never run it from the same process that places orders.