Sample selection: record who drops out at each filter
A dataset’s sample is shaped by every eligibility and cleaning filter, not just the first company list. Record who drops out at each stage before AI describes the result as representative. A complete numeric table can still exclude the entities most relevant to the question if the selection process is hidden.
The Census Bureau’s SUSB methodology documents coverage and exclusions for a specific business-statistics program. It illustrates why a published dataset’s population must be read from its methodology. It does not validate the company-screening rules proposed in this worksheet.
Write the target before the filters
State the population, period, unit and question. A study of all eligible companies differs from a study of companies with three consecutive readable reports. The latter may be useful, but its availability requirement changes the population and should be named in the conclusion.
List each filter in execution order, including identity matching, document availability, numeric completeness, unit compatibility and analytical exclusions. Preserve membership before and after each stage. If an entity fails several rules, retain all applicable reasons while assigning one primary removal stage for the count bridge.
Do not let AI add implicit screens during extraction. A model may return only easy-to-read rows and omit ambiguous tables. Treat extraction failure as a visible processing outcome, not evidence that the company fails an economic criterion. This prevents technical convenience from becoming an undisclosed sample rule.
Hypothetical shrinking sample
Assume a fictional target population contains 200 companies. Identity matching retains 190. A three-report availability requirement retains 150, compatible metric definitions retain 120, and a final completeness rule retains 90. The reported analysis covers 45% of the original target, not all 200.
Suppose most of the 40 companies lost at the report-availability stage are recent entrants. The final sample then favors longer-established firms. That pattern does not prove the estimate is biased in a known direction, but it limits a claim about all entrants and incumbents.
Now change the rule to allow two reports. The resulting sample and estimate may change. Report that as a specification comparison rather than selecting the version with the preferred outcome. All counts are hypothetical, and no actual screening experiment is claimed.
Selection-funnel worksheet
| Stage | Remaining | Evidence question |
|---|---|---|
| Target universe | 200 | Does membership match the question? |
| Identity resolved | 190 | Who failed matching, and why? |
| Reports available | 150 | Is availability associated with company type? |
| Metric compatible | 120 | Which definitions caused exclusions? |
| Required cells complete | 90 | Are missing values selectively concentrated? |
For each stage, calculate remaining share of the original target and loss relative to the preceding stage. These are different denominators. Keep a row-level audit file so the waterfall can be reproduced and a company can be restored if a processing error is fixed.
Investigate selection rather than conceal it
Compare included and excluded groups on variables available for both, such as age, size category or market under documented definitions. Avoid comparing a characteristic that itself was only extracted for survivors. When excluded entities lack the necessary information, report that limitation instead of assuming similarity.
Weighting is not an automatic remedy. A weight requires a defensible target and assumptions about coverage; it cannot manufacture unobserved outcomes. If the final sample is too selective for the original question, narrow the conclusion to the analyzed population or redesign the collection process.
The final memo should name the target, final sample, major loss stages and sensitivity to consequential filters. AI can summarize the funnel after it is constructed, but it should not label the result representative merely because the retained rows are internally consistent.
FAQ
Is a large final sample necessarily representative?
No. Selection mechanisms and excluded groups matter in addition to the retained count.
Should unreadable tables be treated as economic exclusions?
No. Record processing failures separately and investigate whether improving extraction changes the sample.