Download historical stock prices and check coverage
Bulk historical stock data are complete only relative to a declared request and expected coverage. A directory containing one file per ticker can still hide a truncated response, a failed symbol or missing dates inside a range. Keep a request ledger, assess each response independently and generate a retry list for unresolved gaps. Do not infer completeness from the fact that a download process ended without an exception.
The worked example simulates three provider responses for three explicitly supplied sessions. It executes coverage classification and duplicate-safe retry merging using only Python’s standard library. It makes no network requests, proves no vendor entitlement and does not demonstrate a CoTrading download service. Actual market calendars, credentials, licensing and source coverage require separate checks.
Define the request before sending batches
The request ledger needs a market-qualified symbol, start and end dates, frequency, timezone or session convention, price adjustment basis and source identifier. Decide whether the endpoints are inclusive. Save these choices with a request version so a later retry cannot silently switch from raw to adjusted bars or from daily to intraday observations.
Check provider parameters against its current documentation. For example, the Alpha Vantage daily-series documentation distinguishes compact responses from fuller histories and notes access restrictions. A successful response can therefore contain a shorter span than the researcher expected. Requesting a long date interval in a local wrapper does not guarantee that the upstream endpoint honored it.
Use bounded batches and preserve a status for every requested symbol. Avoid a broad exception handler that skips failures and continues without a ledger entry. Record request and response timestamps, error categories, attempt count and the identifier of any saved response. Keep credentials out of filenames and research reports. An expired entitlement and a transient connection failure should lead to different next actions.
Establish an expected-session reference
Our synthetic reference is 2026-10-19, 2026-10-20 and 2026-10-21. These are an explicitly supplied fixture list, not a generated exchange calendar. The request covers AAA, BBB and CCC. AAA returns all three dates, BBB returns only the last two and CCC fails with no response.
In real work, derive expected sessions from the correct market calendar and then adjust for the security’s known listing interval and documented suspensions where appropriate. A weekend-based business-day generator does not capture exchange holidays. A recently listed company cannot supply prices from before listing; that is a structural coverage limitation, not a gap to retry forever.
Python’s datetime documentation supports explicit date and timezone handling, but it does not supply exchange trading calendars. Treat calendar acquisition as a separate input. Keep “market expected a session,” “stock was eligible to trade” and “provider returned a bar” as different observations rather than manufacturing a bar for every calendar date.
Classify each response with dates and counts
For each symbol, collect unique returned dates and compare them with the expected set. Save actual first date, actual last date, unique row count, missing dates and out-of-range dates. Endpoints alone are insufficient: a response can include the first and last dates while missing a date in the middle.
| Symbol | Initial returned dates | Status | Missing sessions |
|---|---|---|---|
| AAA | 19, 20, 21 October | success | none |
| BBB | 20, 21 October | partial | 19 October |
| CCC | no response | failed | all three |
The distinction between partial and failed matters. BBB has usable observations that should be preserved while investigating the missing first session. CCC has no acquired history. Do not place either in the success bucket merely because a file with its symbol exists.
for symbol, response in responses.items():
store[symbol] = set(response or [])
missing = sorted(set(sessions) - store[symbol])
status = "failed" if response is None else "partial" if missing else "success"
coverage[symbol] = {"status": status,
"rows": len(store[symbol]), "missing": missing}
if missing:
retry[symbol] = missingThe complete example is docs/growth/research/round-5/examples/R529.py. Its executed initial row counts are AAA 3, BBB 2 and CCC 0. The retry list contains only BBB’s 19 October observation and CCC’s three missing dates. The script simulates successful gap responses, merges them twice and asserts that every symbol still has exactly three unique dates.
Download the complete example. Use Python 3.12+ with the standard library only; all inputs are synthetic. After downloading, run python R529.py.
Merge retries without hiding conflicting observations
The fixture uses date sets because it tests coverage only. A real price store should key observations by symbol, date, frequency, adjustment basis and source version as needed. Identical repeated bars can be recognized as duplicates. Conflicting bars need a revision decision, not silent set deduplication that throws away prices.
Write new data to a staged artifact, validate it and then publish the new complete version with its coverage table. Keep the old version available for runs that already referenced it. If the retry fails again, leave the initial good rows intact and update only the unresolved status and attempt history. Never mark a download complete just because the retry queue was processed.
Bound retry attempts and elapsed time. Transient failures can be retried with a provider-appropriate delay; unsupported symbols, permissions failures or pre-listing periods need investigation. A run should finish with explicit partial results when the budget is exhausted. This is a successful control outcome even though the requested dataset remains incomplete.
Test a short-history counterexample
Suppose BBB’s missing 19 October is later confirmed to precede its listing. The acquisition process may be complete for the available history, yet the original requested coverage is still not met. Record a structural limitation and the actual supported start date. Do not pad the missing session with the next close or repeatedly fetch an impossible observation.
If instead the endpoint used a compact response, change the permitted request parameters and download the missing range under a new attempt record. The symptom, a short history, looks similar; the cause and repair differ. This is why coverage classification should not jump directly from missing dates to an automatic retry policy without checking source and security metadata.
Hand off files with their unresolved boundaries
The final package includes per-symbol data artifacts, the request ledger, a coverage matrix and a retry or investigation list. Reconcile every requested symbol to exactly one current status. Compare the merged row count with the union of old and newly accepted keys, then inspect several repaired rows against their saved responses.
Downstream calculations should declare whether partial coverage is acceptable. A three-year strategy comparison cannot quietly include one stock with only two months of history. Keep different markets and currencies in appropriate acquisition groups, and verify source support separately. This workflow improves the reliability of obtaining data; it does not guarantee complete global history or certify that the resulting bars are suitable for every backtest.
Preserve a retry ledger that can be replayed
Each retry item should identify the original request, unresolved dates, latest failure category and the next permitted action. Mark a repaired gap only after the new observation passes the same schema and price-basis checks as the initial response. A response containing the requested date but an error string in the close field has not repaired the dataset.
For CCC, the synthetic success branch provides all three missing dates. A real rerun might repair only one, leaving two unresolved. Recompute coverage from accepted stored keys and carry those remaining dates forward, rather than resetting the entire request or assuming all retry items succeeded.
Archive the final ledger together with the files. When source coverage or calendar assumptions change later, it should be possible to replay the classification against the retained responses without redownloading unrelated successful observations.
For overlapping batches, count the union of accepted keys rather than summing file lengths. A revised price is a value revision, not an additional session.
FAQ
Is matching the first and last date enough?
No. Check internal expected-session gaps and unique row counts as well.
Should every missing session be filled with the last close?
No. That creates synthetic observations. Establish the cause and apply only a documented downstream policy if needed.