Search precision: classify false positives before summarizing
Search precision measures how many returned items satisfy a stated relevance rule. Audit false positives before asking an AI tool to summarize the results: an accurate summary of irrelevant documents still answers the wrong research question. Classify the reason for each mistaken inclusion so that you can change the search rather than merely shorten its output.
Set the relevance rule before looking at results
Suppose the question is whether an invented software company’s existing customers expanded contracted spending during the last fiscal year. A document is relevant when it supplies evidence about that company’s existing-customer spending, with an identifiable period and customer definition. General market optimism, a similarly named issuer and a new-customer announcement do not satisfy that rule.
The scikit-learn model evaluation documentation defines precision from true positives and false positives. Applying that arithmetic to a manually labeled research pool is straightforward; deciding what counts as relevant remains the researcher’s responsibility. This example is an editorial evaluation, not a benchmark of a particular search product.
A hypothetical twelve-result review
Imagine the first twelve results contain five relevant records and seven false positives. The precision of this reviewed set is 5 / 12, about 41.7%. This is precision for this query, rule and cutoff. It does not describe all possible queries or tell you how many relevant disclosures were missed outside the returned set.
| Classification | Count | Why it fails the question | Possible correction |
|---|---|---|---|
| Relevant evidence | 5 | Correct issuer, period and customer measure | Keep and reconcile |
| Wrong issuer | 2 | Similar company name or ticker | Use legal name and identifier |
| Wrong period | 2 | Older reporting window | Apply a verified date restriction |
| Wrong metric | 2 | New sales rather than existing spending | Search the exact metric definition |
| Commentary only | 1 | Repeats an unsupported assertion | Find the underlying disclosure |
Review all twelve before changing the query. If you remove each inconvenient result after seeing its conclusion, precision can become a justification for cherry-picking. A relevant document with unfavorable evidence is still relevant. Label uncertainty separately when the document lacks enough context to establish the measure.
Match the correction to the failure
An issuer collision calls for identity resolution, not a longer prompt. Compare the legal entity, exchange, business description and identifiers. A temporal mismatch calls for the reporting period as well as the document publication date: a newly posted file may discuss old operations. A metric mismatch requires checking the underlying denominator, not simply adding the word growth.
For commentary, distinguish relevance from evidential strength. A transcript repeating management’s own estimate might be relevant to what management claimed but insufficient to establish customer behavior. You can retain it as a lead with a lower evidential role. Do not silently redefine the relevance rule so that every weak source disappears from the evaluation.
Compare searches at a fixed review budget
Run an alternative query against the same document collection, relevance rule and result cutoff. Record its relevant count, false-positive mix and any newly recovered evidence. If you compare twelve results from one query with three handpicked results from another, the apparent precision gain is difficult to interpret. Removing the longest documents also changes the inspection budget.
Higher precision alone is not enough. A narrow query can return only one obvious disclosure and miss the footnote that changes the answer. Pair the review with a separate omission audit. Keep a small set of known relevant records to see whether entity filters or metric-specific wording excluded them, while acknowledging that this reference set cannot prove complete recall.
Produce a usable search record
Save the query, collection boundary, time of search, result cutoff, relevance definition and labeled results. Then show the revised query and the observed tradeoff. The output should help another researcher reproduce the triage, including disagreements. Do not replace that record with a vague claim that the AI search was improved.
FAQ
Is a contradictory disclosure a false positive?
No. Relevance concerns whether it addresses the question. Contradictory evidence can be the most important result in the set.
Can the AI assign all relevance labels?
It can propose labels, but a researcher must inspect disputed cases and a sample of apparent successes. Otherwise the evaluation can reproduce the same misunderstanding that caused the search error.