Conflicting sources: make AI expose the disagreement
When sources disagree, ask AI to expose the exact dimensions of disagreement rather than synthesize a single smooth answer. Different dates, entities, definitions, or revisions can produce different numbers without a true contradiction. A genuine contradiction should remain visible until evidence resolves it.
The NIST AI Risk Management Framework provides a general approach to managing AI-related risks. The conflict matrix below is an editorial application for research, not a NIST-prescribed investment method or proof that a particular model resolves disagreements reliably.
Build a conflict matrix
Assign each source an identifier and preserve its original statement. Record the entity, metric definition, period, publication date, version, and evidence type. Separate issuer statements, regulatory records, derived calculations, and commentary. Do not let an AI paraphrase erase the difference between a source’s direct observation and its explanation.
Then formulate the smallest disputed proposition. “Sources disagree about performance” is too broad. “One source reports quarterly revenue for the consolidated group, while another reports annual revenue for one segment” is a scope mismatch that can be investigated.
| Conflict dimension | Diagnostic question |
|---|---|
| Identity | Same entity, security, or contract? |
| Time | Same observation period and publication cutoff? |
| Definition | Same metric and included categories? |
| Version | Was one statement corrected or restated? |
| Evidence type | Observed fact, management claim, or inference? |
| Residual | What disagreement remains after alignment? |
The matrix should preserve unresolved cells. A blank explanation is better than a fabricated reconciliation that nobody can trace to evidence.
Hypothetical revenue disagreement
Assume a fictional issuer’s early statement reports revenue of 100, while a later official document reports 96 for what appears to be the same period. AI might average them to 98 or choose the more recent number without explanation. Neither action resolves the discrepancy.
Inspect whether the later document restates the earlier figure, changes the reporting boundary, or uses a different definition. If a correction is explicit, retain the original historical statement and label 96 as the corrected figure under the stated scope. If no relationship is established, keep the two observations and the unresolved difference.
Now suppose a third source repeats 100 from the early statement. That is not a second independent measurement if it relies on the first source. Record the dependency rather than counting two votes for 100. All numbers and entities in this example are hypothetical.
Use source priority for a specific purpose
Primary sources usually help establish what an organization disclosed or what a rule states. They do not automatically settle every causal question. An issuer may be the authoritative source for its filed number while an independent analysis challenges management’s explanation. State the question before assigning evidential weight.
Ask AI to propose possible explanations with each explicitly labeled as a hypothesis and a corresponding check. For example, a suspected unit mismatch should lead to inspecting table headers; a suspected revision should lead to finding the correction notice. Possibility lists are useful only when they generate discriminating work.
Do not select the explanation that makes the memo easiest to finish. Test it against the original documents. If a source cannot be accessed, disclose that limitation and avoid claiming to have reconciled it through another model’s confident summary.
Decide what the memo can say
After alignment, classify the outcome as resolved definition difference, confirmed revision, genuine unresolved contradiction, or insufficient evidence. The final text should reflect that status. A decision-sensitive unresolved contradiction may require withholding the affected conclusion while other parts of the memo proceed.
Common mistakes include averaging incompatible values, choosing the newest source without reading its scope, treating repeated reports as independent, and rewriting uncertainty into consensus. This method is unsuitable for producing a single definitive number when the relevant evidence remains genuinely inconsistent. Its value is an inspectable account of disagreement and the next check needed, not a forced answer.
FAQ
Should the newest source always replace older information?
No. First establish whether it covers the same question and explicitly revises the earlier statement. Preserve historical knowledge where it matters.
What if the disagreement cannot be resolved?
Report the competing statements, their scopes, and the material uncertainty. Narrow or withhold conclusions that depend on choosing an unsupported winner.