Human review for AI research: assign checks by consequence
Assign human review according to the consequence of an error and the difficulty of correcting its downstream use. A research memo’s central fact, a material calculation, and a stylistic transition do not need identical review effort. The objective is a specific review assignment with visible completion, not a vague statement that a human was involved.
The NIST Generative AI Profile discusses risk management for generative AI. This page offers a practical editorial review allocation, rather than claiming NIST endorses a particular investment workflow or that human presence alone eliminates errors.
Define the use before the reviewer
Write who will use the memo and what they may do with it. A private question list has different consequences from a published factual claim or an analysis used in a capital decision. Review depth should reflect that use, including the ability to withdraw or correct the output.
Identify the claims on which the conclusion depends. Give those claims an owner and a concrete check. “Review the numbers” is less useful than “verify the revenue inputs against the table, recompute the ratio, and confirm the period labels.” A reviewer should know what evidence would justify completion.
| Item | Review action | Completion evidence |
|---|---|---|
| Central factual claim | Open primary source and check scope | Locator and status |
| Material calculation | Recompute from verified inputs | Formula and matched result |
| Uncertain attribution | Check identity and alternatives | Bounded label or unresolved status |
| Sensitive content | Check necessity and access conditions | Approved data scope |
| Narrative | Check attribution and uncertainty wording | Corrected sentence |
The table is an allocation tool, not a substitute for judgment about unfamiliar risks.
Hypothetical two-reviewer memo
Assume a fictional memo compares two businesses and concludes one has stronger cash generation. Reviewer A checks source values and discovers that one cash-flow figure is quarterly while the other is annual. Reviewer B was assigned arithmetic and wording. Without explicit task boundaries, both might assume the other checked period compatibility.
A better assignment makes compatibility a named gate before arithmetic. Once the mismatch is found, the conclusion is reopened, the ratio is withheld, and the memo requests comparable inputs. The review has value because it changes what can be claimed, not because two people signed the document.
Now suppose a low-impact background paragraph has a minor wording issue. It can be corrected without repeating the entire numerical review if the factual dependencies are unchanged. This hypothetical example illustrates proportional review rather than a universal staffing rule.
Review unfamiliar material honestly
A person cannot verify a technical or legal claim merely by reading it carefully. If the consequence is material and the assigned reviewer lacks the relevant competence, narrow the conclusion or seek appropriate expertise. Do not turn a person’s name into an invented specialist endorsement.
Require unresolved issues to remain visible. A reviewer can complete a task by confirming that evidence is unavailable and the claim must be removed. Completion does not always mean approval. Distinguish checked, corrected, unresolved, and excluded so that the release decision reflects the actual outcome.
For AI-assisted work, retain what the AI proposed and what the reviewer changed when that distinction matters to the process. Avoid claiming every word was independently verified if the review sampled only selected assertions. Describe the actual scope of review in ordinary language.
Reopen review when dependencies change
A revised source, corrected input, new deployment, or changed intended audience may invalidate a prior review. Tie completion to the reviewed version and maintain a short trigger list. Rechecking should follow changed dependencies rather than ritualistically rereading everything after every cosmetic edit.
The method cannot guarantee that all reviewers detect all errors. Shared assumptions and incomplete evidence can survive multiple reviews. Common mistakes include undefined signoff, assigning the same narrow check twice while omitting compatibility, concealing unresolved items, and using a reviewer title beyond their actual expertise. A useful release note states what was checked, what changed, and what remains unknown.
FAQ
Does adding a human reviewer guarantee correctness?
No. The task, evidence access, competence, and completion criteria matter. Human involvement without a defined check can leave the same error intact.
Must every edit restart the entire review?
No. Reopen checks affected by changed facts, calculations, or use. A cosmetic change with unchanged dependencies generally needs a narrower check.