Update stock prices with overlap and revision checks
An incremental stock data update should refetch a small overlap, compare observations by key and separate revised bars from new completed sessions. Starting strictly after the last stored date can miss a recent provider correction. Appending every returned row can duplicate dates or mix an unfinished daily bar into sealed history. The useful output is an updated table plus explicit revision, addition and provisional records.
This guide implements that merge in memory with synthetic daily closes and volumes. It does not call a provider, run a scheduler or verify a CoTrading update service. The same process needs separate calendars, session conventions and source contracts for each actual market. A field named complete in this fixture is a supplied control input, not proof that a real feed identifies final bars correctly.
Determine the last valid completed observation
Inspect the stored table before planning the request. Use the latest accepted completed date for the relevant stock, frequency, price basis and source. Exclude provisional observations from that anchor. Retain source revision and retrieval information separately from the observation date, so a later correction can be traced without pretending it is a new trading session.
The fixture’s stored sessions are 19, 20 and 21 October 2026, with closes 10, 11 and 12 and volume 100 each. All three are marked complete. A supplied session list also contains 22 and 23 October. This is a simulated calendar and simulated acquisition state, not a claim about a live request made at the current time.
Choose two stored sessions of overlap, meaning refetch from 20 October inclusive. Find that start in the supplied session sequence rather than subtracting a calendar day and assuming every weekday trades. Real holidays, suspensions and source coverage require appropriate evidence. Save the planned start and end with the request so retries replay the same scope.
Fix the source contract before merging
The incoming observations must match the stock, frequency, currency, price basis and session convention of the stored table. The Alpha Vantage documentation illustrates how raw and adjusted data or extended-session settings can differ. A local merge key of date alone is safe only after those broader dimensions are fixed for the batch.
Investor.gov’s closing-price explanation distinguishes regular-session closes from some after-hours reporting conventions. An observation arriving later in the evening is not automatically a correction to the intended regular close. Verify the source’s finality and session rules before labeling a row complete.
The fixture assumes these upstream checks have supplied valid complete flags. It does not parse timestamps or independently decide exchange-session finality. In a real updater, require a documented market calendar and provider policy, and isolate uncertain bars until that check succeeds. A number changing during a session is an evolving provisional observation, not a historical revision.
Compare overlapping and new keys
Incoming 20 October is unchanged at 11. Incoming 21 October corrects close 12 to 12.5. The completed 22 October close is 13. The 23 October close 13.2 and volume 70 are marked unfinished and must stay outside sealed history.
if not row["complete"]:
provisional[day] = dict(row)
continue
if day not in result:
log["added"].append(day)
elif row != result[day]:
log["revised"].append({"date": day,
"before": result[day], "after": row})
else:
log["unchanged"].append(day)
result[day] = dict(row)Before that loop, the full fixture rejects duplicate response dates and dates outside the declared request. An unordered response can be sorted for presentation, but conflicting duplicate keys need a precedence decision; silently keeping the final occurrence would let transport order choose the accepted history.
The implementation begins from an independent copy of the stored data. Python’s copy reference explains why assignment alone would leave nested mutable objects shared. Keeping the old table intact allows the revision log’s before values to remain genuine evidence after the merge.
Inspect the actual first and repeated update
Download the complete example. Use Python 3.12+, the standard library and synthetic responses; run python R530.py. The source is docs/growth/research/round-5/examples/R530.py.
The executed first update reports overlap start 2026-10-20, one unchanged date, one revised date and one added date. Sealed history has four rows. The revision record preserves 21 October close 12 before and 12.5 after. Provisional dates contain only 23 October, which is absent from the sealed table.
The script merges the identical response again into the updated table. The second addition list and revision list are both empty, and the resulting sealed table equals the first updated table. Provisional storage remains separate. Assertions also verify that the original table still has close 12 on 21 October and that an injected duplicate response date is rejected.
Idempotence here concerns accepted data state under the same request and response. A second attempt can still have a new transport log, but it should not create another logical bar or another numerical revision. Do not confuse recording an attempt with duplicating its underlying observation.
Separate current-bar evolution from historical correction
The unfinished 23 October bar is the different-mechanism counterexample. It has a valid date and numeric close, yet it is not a completed daily observation. Blindly selecting the maximum response date as the next anchor would skip its later final value or make the stored daily history change without a provisional label.
Keep provisional rows in their own structure and promote a row only after finality checks. When the source later provides its completed version, compare it under the declared policy and add that date once. Do not overwrite sealed history with a provisional observation merely because its retrieval timestamp is newer.
The 21 October change is instead a correction to an already completed observation. Preserve both values and the new source reference. A revised raw close should reopen downstream computations that used that value. A changed adjusted series may revise a longer history around a corporate action; a short overlap alone cannot guarantee that all such changes are captured.
Publish the update as one consistent version
The runnable example prints structures rather than modifying production files. A file-backed updater should stage the new table and logs, validate counts and values, then publish a consistent version. Avoid exposing half the new table while the revision log still describes the previous version. Retain the old version for rollback and for research runs that already referenced it.
Reconcile old sealed rows plus new accepted keys with the resulting count; revisions do not increase row count. Rejected or provisional records need their own counts. Inspect the corrected bar and appended bar against saved responses before declaring the update complete.
For the next newly planned job, recompute overlap from the latest sealed date. Replaying the present request deliberately keeps its original overlap start. Define a separate wider reconciliation policy for corrections older than the overlap, because this short-window method cannot discover unseen older revisions. That limitation is a reason to document the update coverage, not to claim a complete historical consistency service.
Deliver the request window, accepted table, revision log, added-date list, provisional table and repeat-run checks. These concrete outputs maintain an ongoing stock dataset while making its changes visible. They do not establish that scheduling is running, that every provider revision is detected or that downstream investment calculations are correct without their own rerun and review.
Review the revision before publishing downstream freshness
Attach a source-response locator to each changed row and distinguish a numerical revision from a retrieval-metadata refresh. A new download timestamp alone should not produce a new price revision. If the numeric record is unchanged, retain the successful comparison without inflating the revision count.
For a file-backed process, compare the staged table with the proposed log before making it visible. Every addition should introduce one new accepted key, every revision should identify an existing key and every provisional row should remain outside sealed history. A mismatch between those structures should stop publication even if the final row count looks plausible.
After publication, mark downstream calculations using a revised input as requiring recomputation until their own checks finish. A fresh source table does not automatically make an older chart or strategy report fresh. Keeping that state explicit connects ongoing data maintenance to its real consumers without pretending that this narrow merge function already runs the entire research pipeline.
FAQ
Why refetch dates already stored?
The overlap can reveal recent corrections and validate unchanged observations. An append-only request cannot compare values it never retrieves again.
Does the newest returned date become the new anchor?
Only if it is accepted as a completed observation under the intended session and source contract.