Handle missing stock screener data with a repair queue
Missing stock screening data should affect execution, not merely appear as a note below a finished candidate list. Evaluate each agreed condition as true, false or unknown, then separate passing stocks, known failures and unresolved stocks. Generate field-specific repair work and rerun the rows whose accepted inputs change. The result is an executable recovery process rather than a collection of missing-value labels.
This guide uses four synthetic stocks and two fixed numerical conditions. It actually creates a repair queue, completes one observation and checks the resulting state transition. It does not retrieve a disclosure, retry a real provider or demonstrate a current CoTrading data-quality service. Coverage and field availability must be checked independently for each market and source.
Keep the value separate from its usability
A field record should contain an observed value and a status, plus the source and date metadata needed to determine whether that value is usable. A stale observation can contain a perfectly parseable number. A provider error may contain no number even when the company has disclosed the information elsewhere. An undisclosed item needs a different investigation from a temporary connection failure.
For the fixture, each field is represented by (value, status). Only status observed with a nonmissing value enters the numerical predicate. stale, provider_error and not_disclosed become unknown at the condition level. These statuses are declared synthetic inputs, not inferred automatically from an empty response. A production acquisition layer must establish their meaning from its actual response and disclosure evidence.
Do not use ordinary truthiness to determine field availability. The Python expression reference explains that None, zero and empty values can all behave as false in Boolean contexts. Measured growth of zero and unavailable growth are different observations, so explicitly check status and missingness before comparing the number.
Define three-state conjunction
Our rules require positive PE below 20 and revenue growth at least 10%, stored as ratio 0.10. For their conjunction, any observed false condition makes the overall result fail. If there is no false but at least one unknown, the result is unknown. Only two observed true conditions pass. This policy preserves a known disqualifier even when another field is missing.
def evaluate(row):
checks = []
for field, predicate in [("pe", lambda v: 0 < v < 20),
("growth", lambda v: v >= .10)]:
value, state = row[field]
checks.append(predicate(value)
if state == "observed" and value is not None else None)
return "fail" if False in checks else "unknown" if None in checks else "pass"Do not automatically extend this conjunction to arbitrary OR trees. Under an alternative-route rule, one known true branch can sometimes establish a pass despite an unknown alternative. Define that logic separately. The present recovery fixture focuses on two mandatory conditions so its dependency and state transition are easy to inspect.
Run the incomplete input table
AAA has PE 18 and observed growth 0.12. BBB has PE 25 and undisclosed growth. CCC has PE 16 but its growth request failed. DDD has PE 17 and a stored growth value 0.15 that is marked stale. The stale numeric value is deliberately tempting: accepting it would make the screen look more complete than its validity policy allows.
| Stock | PE | Growth value | Growth status | Screening state |
|---|---|---|---|---|
| AAA | 18 | 0.12 | observed | pass |
| BBB | 25 | missing | not disclosed | fail |
| CCC | 16 | missing | provider error | unknown |
| DDD | 17 | 0.15 | stale | unknown |
BBB is a known failure because its PE violates a mandatory condition. Its disclosure gap still belongs in the data-quality queue, but it need not receive the same research priority as CCC, whose candidacy is unresolved. Screening state and repair priority are separate fields; keeping both avoids mislabeling every incomplete row as unknown.
Produce actions that match the failure mechanism
The fixture creates three work items: confirm BBB’s disclosure situation, retry CCC’s source request and refresh DDD’s stale observation. It does not execute those external actions. Each work item names the stock, field, status and requested action so a data owner can resolve the specific dependency.
For an actual retry, preserve attempt count and the original request. For stale data, define the applicable freshness rule rather than treating any newer timestamp as sufficient. For an undisclosed field, confirm whether the metric is genuinely unavailable or merely absent from the selected source. Avoid asking a language model to invent a plausible number to make a required column complete.
The queue may include gaps on already failed stocks, as BBB demonstrates. That is useful for full data maintenance, but a limited research team can prioritize unresolved candidates first. Record that prioritization separately from the truth of the numerical conditions. Deferring BBB’s repair does not mean its missing growth has become an observed failure.
Complete one field and rerun its dependency
Download the complete example. It uses Python 3.12+, the standard library and synthetic inputs. Run python R521.py; the authoring file is docs/growth/research/round-5/examples/R521.py.
The script makes a separate copy of the input, supplies CCC growth 0.20 with observed status, and reevaluates CCC only. Actual output changes CCC from unknown to pass. AAA stays pass, BBB stays fail and DDD stays unknown. Assertions reconcile four stocks before and after, identify CCC as the only changed member and verify that the original CCC record remains missing with provider_error status.
The Python copy reference distinguishes object assignment from independent copying. The fixture uses a deep copy because its rows contain nested mutable structures. Preserving the old input lets a reviewer reconstruct the actual repair rather than discover that both “before” and “after” now point to the same modified object.
This local rerun is valid because the thresholds are fixed and rows are independent. If a repaired field contributes to an industry median, cross-sectional rank or another shared statistic, the dependency expands to other stocks. Compute the affected set from the actual rule rather than applying “only one row” as a universal optimization.
Challenge the stale-value counterexample
DDD’s 0.15 would satisfy the growth predicate if it were usable. The fixture withholds it because status is stale. Copying the numeric value into an observed slot without new evidence would convert unknown to pass through a metadata lie, not a real repair. The repair needs a valid observation or a deliberately revised freshness policy with its own rule version.
CCC’s provider error is a different mechanism: the field may exist, and another attempt can obtain it. BBB’s undisclosed status can require confirmation rather than repeated requests. The same null value therefore does not justify one generic retry action. Preserve the reason that made the field unusable and update it only with supporting evidence.
Close the recovery loop with a transition report
Deliver the initial states, repair queue, accepted replacement input, rerun stock list and before-after changes. Record the new source and date when real data replace fixture values. If a repair produces growth below the floor, the correct transition is unknown to fail; repair success means the condition became evaluable, not that a stock entered the candidate set.
Check that no unrelated row changed, no input vanished and no unresolved field was replaced by zero. Keep unsuccessful repairs open with their latest evidence rather than marking the entire queue done because a worker attempted each item. Reopen the process after source corrections, freshness expiry or new disclosure coverage. This turns missing-data handling into a reproducible screening operation with bounded, explainable follow-up work.
Accept the repair evidence, not just a new cell
Before applying a replacement, compare its security, field, period, unit and availability date with the blocked dependency. A new number for a different fiscal year does not complete the requested observation. A profitability measure cannot substitute for missing revenue growth unless the research rule itself is deliberately revised.
Record who or which acquisition step accepted the replacement and which source supports the state becoming observed. This is useful even in a small file-based workflow: it distinguishes a genuine acquisition from an accidental manual status edit. The repair record should identify the old value and status as well as the new pair.
After the targeted rerun, reconcile both stock counts and repair-item counts. A stock can have several unusable fields, so one completed item may leave its screening outcome unknown. Conversely, a known false condition can settle the screening decision before all maintenance work is finished. Preserve these different completion levels rather than assigning one success flag to the entire recovery operation.
FAQ
Should any missing field make the whole stock unknown?
Not under this AND policy when another mandatory condition is already false. Preserve both the known failure and its data gap.
Does a successful repair have to create a candidate?
No. It can establish a known failure. The goal is a supported decision, not a larger candidate list.