Estimate backtests: separate outcomes from earlier bias
An estimate that differs from its eventual settlement was not necessarily unreasonable when made. Retrospective review asks what the earlier information supported, what changed afterward, and whether a repeated directional pattern says something useful about current estimation. It does not turn every forecast miss into a proved accounting error or fraud.
This guide concerns PCAOB issuer audits, using AS 2401 and AS 2501 as displayed on October 25, 2026. The case uses fictional claim liabilities. It is a public-research adaptation of an evidence discipline; the worksheet does not imply that an investor has access to confidential legal files or the auditor’s estimate testing.
Freeze the earlier information set
AS 2401 .63–.65 addresses retrospective review of estimates for possible management bias, particularly selected estimates with assessed fraud risk. It also distinguishes learning for current estimates from calling prior judgments into question merely through hindsight.
Create a dated evidence file for each estimate: measurement date, approval date, method, inputs, reasonable alternatives considered and information available at that time. Keep the later settlement document in a separate layer. A revised spreadsheet created after settlement is not a reliable substitute for the original forecast record.
Distinguish when a condition existed from when the company or auditor learned about it. A later document can reveal an earlier condition; another later document can describe a genuinely new event. The date on the document alone does not resolve that distinction.
For public research, an annual note may establish the recognized amount but not its detailed assumptions. Mark undisclosed inputs as unknown rather than reconstructing a supposedly contemporaneous model using facts that became known months later.
Work three settled cohorts
All amounts below are hypothetical thousands of currency units. Three comparable claim cohorts were estimated at 10, 12 and 8. Their later settlements were 14, 15 and 11. Assume identical scope, currency and treatment of settlement costs; there are no other measurement differences.
| Cohort | Earlier estimate | Settlement | Outcome minus estimate | Supported later-event effect | Unexplained remainder |
|---|---|---|---|---|---|
| A | 10 | 14 | 4 | 2 | 2 |
| B | 12 | 15 | 3 | 1 | 2 |
| C | 8 | 11 | 3 | 3 | 0 |
| Total | 30 | 40 | 10 | 6 | 4 |
The unadjusted outcome gap is 40 − 30 = 10. Assume separate evidence supports effects of genuinely later events totaling 6, allocated 2, 1 and 3. The remaining difference is 10 − 6 = 4.
That 4 is an investigation remainder, not automatically an earlier misstatement or an intentional understatement. The table does not claim that removing hindsight produces a uniquely correct prior estimate of 34. Estimation uncertainty and evidence quality still matter.
Nor is 6 a balancing plug. Each attributed later event needs a dated description, a causal link to the payment and a supported amount. If only 4 of the proposed 6 is supported, the unexplained remainder becomes 6. The worksheet must change rather than preserving the preferred residual.
Examine the mechanism behind the remainder
For cohort A, ask whether information available at the original date showed a higher claim frequency than management used. For B, inspect whether settlement severity assumptions were supported and whether unfavorable evidence was excluded. For C, the assumed later-event explanation fully accounts for the observed difference, but that does not prove every original assumption was perfect.
A repeated residual in the same direction can justify further testing of current assumptions and incentives. It is stronger evidence when the cohorts are comparable, the causal analysis is independent and alternative explanations have been tested. Three settlements alone do not establish management intent.
Compare favorable and unfavorable outcomes symmetrically. Keeping only underestimates while dropping overestimates creates the pattern being investigated. Conversely, netting unrelated overestimates against a concentrated understatement can hide a particular weak model.
Separate model design, input data and execution. A reasonable method fed incomplete claims data is a different issue from a method that structurally excludes a relevant risk. The remedy and implications for current estimates may differ.
Later evidence must relate to the right conditions
AS 2501 .27–.31 addresses later events or transactions as evidence about conditions at the measurement date and evaluation of bias individually and in aggregate. The passage of time increases the possibility that relevant conditions changed.
Consider a new law enacted after the measurement date that, under the explicitly assumed legal facts, creates a new payment obligation and reflects no earlier obligation. That is a different mechanism from discovering that a preexisting claim was omitted from the original population.
In the first case, the later settlement can change without proving the original estimate wrong. In the second, the newly discovered information may be relevant to the earlier estimate because the underlying condition already existed. The legal classification and timing need actual evidence; a press headline saying “new development” does not decide them.
A natural disaster occurring afterward provides another possible new-event mechanism. But if the claimed loss arose from damage already present at measurement, the explanation must be revisited. Do not classify events by convenient labels.
Avoid a closed-case selection trap
Suppose the original pool also contains a fourth cohort estimated at 20 that is still open. The complete earlier estimate pool was 50, but observed settlements of 40 relate only to the first three cohorts with earlier estimates of 30.
It would be wrong to compare cash settlements of 40 with the whole 50 and announce an overestimate of 10. It would also be wrong to announce that the whole pool has a known understatement of 10. The unresolved cohort’s outcome is unknown.
Retain open, settled, reopened and partially settled states. Settlement speed can be related to severity: straightforward small claims may close first, while difficult claims remain unresolved. A settled-only backtest can therefore describe a selected subset.
For partial payments, identify whether a remaining liability is still outstanding. Payment to date is not the final outcome. Preserve recoveries and defense costs separately when their scope differs from the original estimate.
Use the review to test current estimates
Suppose current claims are estimated at 40. Do not mechanically add the historical residual of 4, or scale it by a convenient percentage, to declare the current liability must be 44. Check whether current claim types, exposures, legal conditions and methods are comparable first.
A useful response might be testing whether the current model includes the information omitted in A or the severity factors questioned in B. Recalculate a conditional scenario with stated inputs, then compare it with the applicable accounting framework and reasonable outcomes. Historical review provides questions and evidence, not a universal uplift formula.
Document who supplied each explanation and whether independent records corroborate it. An unsupported management narrative and a verified legal change should not receive the same evidence status merely because both explain part of a gap.
Execute the review register
| Field | Input to retain | Control |
|---|---|---|
| Earlier estimate | Original amount, method and date | No hindsight rewrite |
| Population | Cohort definition and open cases | No settled-only selection bias |
| Outcome | Final or partial settlement and scope | Comparable amounts |
| Change events | Date, condition and causal amount | No residual plug |
| Remaining difference | Unexplained amount and alternatives | Not automatically error or fraud |
| Current implication | Specific input or method to retest | No mechanical uplift |
| Evidence status | Public, private, corroborated or unknown | Limits visible |
When disclosures conflict, compare reporting basis, gross versus net settlement, currency and cohort scope. A reserve movement note includes new provisions, payments and reversals; it is not automatically a backtest of the original closed cohort.
Update the file when a claim closes, a previously open case reopens, contemporaneous evidence is obtained, or a claimed later-event explanation fails. Preserve the original estimate and each revision date so the analysis distinguishes learning from retrospective rewriting.
Compare a point estimate with an earlier reasonable range
In an independent case, assume the evidence available at measurement supported a reasonable range of 8 to 13 and management recorded 10. Later settlement is 14, including an independently supported effect of 3 from a genuinely new event. The outcome adjusted only for that stated new event is 11.
Both the original 10 and the conditional 11 lie inside the assumed earlier range. Their difference of 1 does not automatically establish an earlier misstatement. Conversely, the fact that a number lies within a range is useful only if that range was itself appropriately supported at the time. A broad range invented after settlement cannot validate the original judgment.
FAQ
Does a settlement above an estimate prove the earlier estimate was wrong?
No. The original information, uncertainty and later changes must be assessed.
Does the unexplained remainder of 4 prove fraud?
No. It identifies a question for investigation and current-estimate testing.