Retrieval misses: audit relevant evidence outside the results
A retrieval miss audit examines relevant evidence that did not appear in the result set. Start with a bounded collection whose relevance you can inspect independently, then compare the returned items with what the collection actually contains. Reviewing only the results can reveal irrelevant hits, but it cannot reveal material the search never showed you.
Freeze the collection and define relevance
Specify the research question, issuer or asset, period and document boundaries. Save the query and any filters. Decide what counts as relevant before reading the result ranking. A document can be important to the company yet irrelevant to this particular question, so use a task-specific criterion rather than a general impression of quality.
The scikit-learn evaluation documentation defines recall through true positives and false negatives. Applied to this editorial audit, that means relevant returned items divided by all relevant items in the checked collection. This is a defined-pool measurement, not a way to claim that every relevant document on the open web has been discovered.
A hypothetical known-pool audit
Assume you assemble twenty original disclosure excerpts about one invented issuer’s revenue contracts. Independent reading identifies ten as relevant to cancellation rights. Your search returns eight excerpts, six relevant and two unrelated. Four relevant excerpts remain outside the result set. Recall within this fixed pool is six divided by ten, or 60 percent.
| Missed item | Candidate reason | How to test the reason |
|---|---|---|
| Contract footnote | Query used only the headline term | Search a documented synonym |
| Earlier amendment | Date filter excluded the file | Re-run with the declared historical window |
| Scanned appendix | Text was not extracted | Compare searchable text with the original image |
| Alternate issuer name | Identity mapping omitted the alias | Test the verified identifier and name |
These reasons are hypotheses until tested. A missed scan should not be blamed on ranking before you establish that its text entered the searchable collection. Keep the original results so that a repaired run does not erase the evidence of the failure. The assumed percentages describe only this example, not a measured capability of any AI product.
Search the complement deliberately
The complement is the material in your bounded collection that was not returned. Inspect it directly, or choose a documented sample when the collection is large. If you inspect only a sample, report sample size and how it was chosen; do not label the observed fraction as the collection’s complete recall. An easy sample of short readable documents can miss the very formats causing retrieval failures.
Include document types, date ranges and language variants likely to behave differently. Keep newly discovered relevant items in an audit ledger with source position and relevance reason. Separate true misses from disagreements about the question’s scope. If a scope rule changes during review, record the change and reassess the entire pool consistently rather than relabeling only inconvenient items.
Repair one stage at a time
A useful repair sequence is ingestion, identity, filters, query vocabulary and ranking. Verify whether each relevant source was actually available to the system before adjusting the prompt. Broader queries may recover missed items while also returning more irrelevant material. Check both consequences instead of declaring improvement solely because the result count increased.
After a change, run the same fixed collection and compare the identities of recovered and newly missed items. Keep a few fresh cases outside the tuning collection for a later check. A search tuned to remember the audit examples may not work equally well on another issuer’s documents. The audit should expose that limit, not obscure it with one aggregate score.
Connect misses to research conclusions
Ask whether any missed source changes a material statement in the memo. Missing a cancellation clause can matter more than missing another repetition of revenue growth. Record the affected claim and the additional review required. A recall figure summarizes retrieval behavior within its pool; it does not measure the financial significance of every omitted item.
FAQ
Can I estimate recall by reading only the top results?
No. That can assess relevance among returned items, but the denominator for recall requires relevant material outside the results or an independently defined reference pool.
Does recovering more documents always improve the workflow?
Not necessarily. Extra hits can add noise or duplicate information. Measure recovered relevant items, remaining misses and the cost of reviewing false positives.