Resume interrupted stock downloads from a verified cache
Resume an interrupted stock download by verifying completed cache entries, preserving attempt counts and requesting only unfinished work. A file’s presence is not a success marker: it may contain a partial response, an obsolete request or corrupted bytes. Define a cache contract and a bounded recovery plan before starting a large stock list.
The example actually writes and reopens files in a newly created temporary demonstration directory. Data requests are synthetic functions, not network calls. It neither reads nor deletes existing user caches, and it does not establish a current CoTrading or Vibe generic downloader. The output remains partial when a stock exhausts its retry budget.
Split work into stable request units
Our universe is AAA, BBB, CCC and DDD, arranged as two batches: AAA with BBB, then CCC with DDD. Each request specifies source, stock, start and end dates, interval, adjustment basis and currency. The fixture asks for two daily dates, 19 and 20 October 2026, with raw USD closes.
Create a deterministic key from the complete request, not the ticker alone. A weekly response or an adjusted series must not satisfy a request for raw daily bars. Source identity and date range matter too: a successful earlier window cannot prove that a later interval has been obtained.
The Python hashlib reference documents SHA-256 used here for request keys and content digests. A digest helps identify bytes and detect a changed cache payload. It does not prove that a provider’s prices are economically correct or that its original response meets every financial-data requirement.
Define what a reusable cache entry contains
Each logical entry has a data file and a separate completion marker. The marker retains the exact request and a digest of the published data bytes. The reader requires both files, matching request metadata, a matching digest, the expected dates and positive finite close values. An incomplete or incompatible entry returns unavailable for reuse.
metadata = json.loads(marker.read_text(encoding="utf-8"))
content = data.read_bytes()
if metadata["request"] != request:
return None
if metadata["sha256"] != hashlib.sha256(content).hexdigest():
return None
rows = json.loads(content)The full fixture also checks its two-date schema. For real OHLCV data, add field, unit, duplicate-key and coverage checks appropriate to that request. A hash validates identity of the retained bytes; it cannot substitute for those checks. A valid JSON response containing an error message must not receive a success marker.
Publish data before the completion marker
Write and close a staged file in the same directory, then reopen and validate that staged payload. Only a valid payload may replace the intended data file; publish the completion marker after that replacement. Invalid payloads raise invalid_cache_payload without a new marker, and an invalid replacement leaves an existing verified data file and marker unchanged. Python’s os.replace documentation describes replacement behavior and its filesystem limitations. This example uses a single process and does not claim a transactional database, concurrent-writer safety or power-loss durability.
Two files are not one atomic transaction. If interruption occurs after the data replacement but before its new marker, the reader should reject a missing or mismatched marker instead of reusing the half-published entry. That conservative check is the recovery mechanism; merely renaming a file does not solve all failure cases.
Persist the attempt counter before making the mock request. An interrupted attempt then consumes budget rather than being forgotten on every restart. Keep attempt history separate from data success: writing the counter is not proof that a usable response was obtained.
Simulate the interruption and resume
Initially AAA is fetched once and committed with a valid marker. BBB’s first synthetic response is written only as a truncated data file, without a marker. This simulates an interrupted batch write; it does not kill a real downloader process. CCC and DDD have not yet completed.
Download the complete example. Use Python 3.12+, the standard library and synthetic inputs; run python R537.py. The authoring source is docs/growth/research/round-5/examples/R537.py.
The first resume reuses AAA, fetches and commits BBB, tries CCC twice and records budget failure, then completes DDD. Total mock request counts are AAA 1, BBB 2, CCC 2 and DDD 1. BBB’s count includes its interrupted first attempt. The per-request maximum is two attempts, so persistent CCC failures cannot create an endless loop.
| Stock | First resume | Second resume |
|---|---|---|
| AAA | reuse verified cache | reuse verified cache |
| BBB | download missing completion | reuse verified cache |
| CCC | failed budget | failed budget, no extra request |
| DDD | download | reuse verified cache |
The second resume reopens the saved attempt ledger. It reuses all three valid entries and makes no additional mock requests, including for CCC. Assertions verify the statuses and unchanged call counts. This tests restart behavior using persisted files, not only an in-memory variable that survives one function call.
Challenge partial files and corrupted content
BBB’s truncated file is the first counterexample. Its filename exists, but it is not reusable without a valid completion marker and schema. Treating existence as success would skip BBB during recovery and leave an apparently complete batch with unusable data.
A separate corruption probe copies a valid entry into a new demonstration subdirectory, changes its payload and verifies that the digest check rejects it. A publication probe attempts a negative-close commit before any marker exists, then attempts the same invalid replacement over a verified entry. It asserts no new marker for the first failure and byte-for-byte preservation of the valid data and marker for the second. Another check requests AAA with weekly frequency or adjusted basis and confirms that neither reuses the raw daily entry.
The tempfile reference documents the creation of the fresh directory used by this test. The script prints its actual path and leaves these demonstration files there; it performs no deletion of preexisting caches. Each execution creates a new isolated location, so a reader can inspect the data, markers and attempt ledger without connecting to a provider.
Preserve failure limits and honest completion
Three usable stocks and one failed stock are not a completed four-stock dataset. Deliver the valid cache inventory, failed-budget item and request scope together. A user can later approve or configure another attempt budget after investigating the cause, but the recovery loop should not reset counters merely because the process restarted.
For a real provider, distinguish transient timeouts from invalid symbols, permissions errors or unsupported historical intervals. Retry only appropriate failures and honor the actual source’s restrictions. The fixture uses one synthetic timeout type and does not simulate every transport or entitlement condition.
Avoid overwriting a verified older entry with an unfinished response under the same logical key. Stage and validate first, retain provenance and make the selected version explicit. If request meaning changes, generate a different key rather than silently relabeling the old bytes.
Deliver a replayable recovery package
Keep request configurations, batch membership, attempt ledger, data files, markers, validation outcomes and unresolved items. Have a reviewer reopen a reused entry and a rejected partial entry from disk. Then replay resume and verify that completed work is not requested or written again.
The cache is not a historical-coverage guarantee. A validated two-date file proves only its declared request and schema. Wider ranges, additional fields and source corrections need their own acquisition and reconciliation rules. This workflow contributes concrete disk persistence, interruption handling and bounded retries to stock data collection while leaving provider truth and downstream strategy correctness to separate checks.
Review cache validity separately from retry status
A stock can exhaust its attempt budget while another stock already has reusable data. The recovery report should preserve both outcomes instead of aborting the entire universe or relabeling the failed item as complete. Reusing a verified entry and declining another request are distinct successful control paths even though only one supplies data.
Keep the request-key definition with the cache format version. If a future implementation adds a session convention or changes numeric serialization, assess whether old entries remain compatible before reusing their success markers. A digest only describes the bytes it covers; it does not announce a changed interpretation of those bytes.
Inspect the new demonstration directory after a run if desired: its data files, completion markers and attempt ledger are actual local artifacts. A real service needs additional decisions about retention, permissions, concurrent updates and durable commits. Those requirements should be tested in its own storage environment rather than inferred from this single-process synthetic recovery example.
FAQ
Is a downloaded filename enough to skip a stock?
No. Validate its request, completion marker, content digest, schema and required coverage first.
Does restarting reset the retry budget?
Not here. Attempt counts are persisted, so repeated restarts cannot silently create unlimited retries.