Reviewer agreement: measure disagreement before adjudication
Measure reviewer disagreement before resolving it. Two people reaching the same label is evidence about their agreement, not proof that the label is correct. Preserve the original independent decisions, build a cross-tabulation and interpret chance-adjusted agreement alongside class prevalence and the consequences of different mistakes.
Keep independent labels separate from consensus
Imagine two researchers assess whether disclosure excerpts support a narrowly defined accounting claim. Give them the same written rule and source material, but collect their initial labels separately. If they discuss every example before recording a decision, the resulting consensus cannot tell you how consistently the rule works between reviewers.
The scikit-learn documentation for Cohen’s kappa gives a chance-adjusted agreement statistic based on observed agreement and the reviewers’ marginal label frequencies. It is a measure of agreement between two annotators, not a certification of factual accuracy or expertise. The research example below uses invented labels.
A hypothetical fifty-excerpt audit
Suppose Reviewer A labels 20 of 50 excerpts as supported and Reviewer B labels 26 as supported. Their joint decisions produce the following table. Neither reviewer is automatically designated the reference truth, so the off-diagonal cells should be called disagreements rather than classifier false positives or false negatives.
| Reviewer A / Reviewer B | Supported | Not supported | Row total |
|---|---|---|---|
| Supported | 18 | 2 | 20 |
| Not supported | 8 | 22 | 30 |
| Column total | 26 | 24 | 50 |
Observed agreement is (18 + 22) / 50 = 0.80. Expected agreement from the marginal frequencies is (20 / 50 × 26 / 50) + (30 / 50 × 24 / 50) = 0.496. Kappa is (0.80 - 0.496) / (1 - 0.496), approximately 0.603. Ten excerpts still require investigation regardless of how appealing that single summary looks.
Inspect the disagreement mechanism
Read the ten disputed excerpts with their original reasons intact. Perhaps one reviewer treats management’s assertion as sufficient support while the other requires the underlying calculation. Another disagreement may reflect an ambiguous reporting period, a missing footnote or a poorly defined label. Those are different failures with different repairs.
Distinguish instruction problems, missing context, legitimate interpretation differences and simple oversight. Revise the rule when necessary and record the change. Do not relabel the original audit silently after a discussion, then present the newly unanimous labels as evidence that the original instructions produced perfect agreement.
Read kappa with the margins
When almost all cases fall into one class, raw agreement can be high simply because both reviewers usually select that class. Kappa adjusts for a particular chance model using marginal frequencies, and its value can be sensitive to those frequencies. A change between datasets can therefore reflect case mix as well as a change in reviewer consistency.
Report the full table, sample size and class distribution. Avoid treating a universal adjective such as excellent as a substitute for a task-specific assessment. Two reviewers agreeing on most routine negatives may still disagree on the few material disclosures that determine an investment memo’s conclusion. Weight the operational consequence separately rather than inventing a financial meaning for a kappa threshold.
Adjudicate without erasing history
Assign disputed cases to a documented adjudication process, using the source and rule rather than seniority alone. Save the initial labels, adjudicated label, reason and evidence locator. If both reviewers made the same mistake, independent agreement will not expose it; include a separate factual check on a sample of apparently agreed cases.
After changing the instructions, evaluate fresh excerpts or clearly label a repeated set as a training exercise. Improvement on a familiar set can reflect memory of the earlier discussion. Keep evaluation and instruction development distinct if you intend to make a broader statement about reliability.
Bound the final claim
A useful report says how often the reviewers initially agreed, where they differed and which decisions changed after source review. It does not claim that agreement proves truth or that one small audit establishes consistency for every language, issuer or document type. Record the population and date so the finding remains interpretable.
FAQ
Is kappa the same as classification accuracy?
No. It compares two sets of labels without making either reviewer the true reference. Accuracy requires a separately justified reference standard.
Should I delete original labels after adjudication?
No. Preserve them with the final decision. Otherwise you lose the evidence needed to diagnose and improve the review process.