AI factual errors: triage invented details systematically
Triage AI factual errors by isolating checkable details before evaluating the narrative. Entity names, dates, document identifiers, quantities, and event descriptions are useful starting points. A fluent research summary should remain a draft until the details carrying its conclusion have been checked against appropriate evidence.
The original SelfCheckGPT paper investigates a consistency-based approach to identifying nonfactual generated statements. It is a research contribution under particular conditions, not proof that repeated agreement establishes truth for every current model. This workflow treats model disagreement as a review lead and primary evidence as the basis for verification.
Make a factual quarantine list
Copy the draft and mark the details that would change the conclusion if wrong. Assign each detail a stable identifier. Keep it outside the approved evidence set until its status is resolved. This prevents a suspected invented date from being copied into several later notes as if repetition had verified it.
Classify the item so the verification method matches the error type. A wrong entity requires identity checking; a calculation requires input and formula checks; a nonexistent report requires document discovery. Asking the same broad prompt again may produce a different story without addressing the underlying uncertainty.
| Item class | Verification task |
|---|---|
| Entity | Confirm exact issuer, legal name, ticker, or contract |
| Document | Locate the actual publication and identifier |
| Event | Confirm occurrence, date, and stated outcome |
| Quantity | Match source value, unit, period, and denominator |
| Explanation | Separate source attribution from analyst inference |
Use statuses such as verified, corrected, unresolved, and removed. Avoid “probably true” as a substitute for specifying what is missing.
Hypothetical invented filing
Assume an AI draft says a fictional company filed a “2026 strategic liquidity report” in April and reduced debt by 30%. The researcher finds no such document in the issuer’s official publication archive. That does not prove no relevant information exists, but it means the cited report has not been established.
Split the sentence into the existence of the document, the date, and the debt change. A later search might locate a quarterly statement with a different title and a 30% decline in a different liability category. That is not permission to retain the original sentence. The correct revision must reflect the actual document, period, and definition.
All entities and figures here are hypothetical. The example illustrates why a partly recognizable number can make an invented source sound credible. Correcting one detail without reopening the surrounding claim can leave the main error intact.
Use disagreement carefully
A second answer can reveal unstable names, shifting dates, or unexplained numbers. Preserve those changes as leads. Agreement across repeated answers is weaker evidence than directly locating the relevant source, especially when the answers may rely on similar training material or on each other’s wording.
Do not resolve two conflicting model answers by majority vote. Instead identify which check would discriminate between them. If they disagree about a filing date, inspect the publication record; if they disagree about a ratio, obtain inputs and calculate it independently. The verification task should become narrower as the uncertainty becomes clearer.
For causal explanations, check whether the source actually attributes the change to the stated cause. If only correlation is visible, label the explanation as a hypothesis. The absence of an invented number does not make an unsupported causal story reliable.
Prioritize by consequence
Check details that support the central conclusion before cosmetic wording. A wrong company identity, period, sign, or denominator can invalidate an entire memo. Once a high-impact error is found, inspect nearby assertions derived from it and update dependent calculations or summaries.
Keep an error log with the original statement, correction, source, and affected notes. This is a process-improvement record, not a claimed model accuracy score. A meaningful score would require a defined evaluation set and consistent judging rules.
The method cannot establish facts whose relevant sources remain unavailable. In those cases, stop the unsupported claim from entering the conclusion. Common mistakes include trusting polished prose, asking for confidence percentages, treating repeated agreement as proof, and correcting a number without checking the entity or period attached to it.
FAQ
Can asking the model whether it hallucinated verify the answer?
No. The response can suggest checks, but the material claims still need suitable external evidence.
Must every sentence receive the same review effort?
No. Prioritize factual details that change the conclusion, then complete the remaining checks required by the memo’s intended use.