Review a failed thesis without judging only the outcome
Review a failed thesis by separating the quality of the original process, the behavior of the proposed mechanism and the eventual outcome. A good process can encounter an unfavorable outcome, and a weak process can get lucky. Neither statement excuses avoidable research errors.
Reconstruct the original question first
Open the dated thesis and evidence set before reading your latest interpretation. Identify what success meant, which assumptions were necessary and when the test was supposed to resolve. If the original record is missing, label the reconstruction as retrospective.
The 10-K provides business, risk and financial information, but only documents available at the original decision date belong in the historical evidence set. Later disclosures can explain the outcome; they cannot prove you should have known their contents earlier.
Hypothetical postmortem: delivery expansion
Assume the thesis expected a distributor’s new region to reach sufficient route density in a year. The region instead generated losses. Later evidence shows that customers ordered less frequently and a competitor lowered prices.
| Review layer | Question | Hypothetical finding |
|---|---|---|
| Process | Was the starting demand estimate supported? | Used customer count without order frequency |
| Mechanism | Did density reduce cost as expected? | Cost improved, but insufficient stops limited savings |
| External change | What changed after the original note? | Competitor pricing weakened revenue per stop |
| Outcome | What actually happened? | Losses exceeded the assumed operating range |
| Improvement | What repeatable change follows? | Require frequency and price sensitivity before regional extrapolation |
Do not force one cause to explain everything. The initial frequency omission is a process defect. The competitor’s later move is a subsequent event. Both can matter, while neither implies that every loss was predictable.
Compare expectation with observation
For each necessary assumption, list the original value or qualitative condition, the observed result and the reason for any comparability issue. Identify whether an invalidation trigger appeared and whether you acted on the research record. A correctly designed warning is not useful if it was repeatedly explained away.
Write one or two specific process changes that can be tested next time. “Be more careful” has no observable implementation. “Require demand frequency before using customer count to estimate route density” is concrete. Avoid adding an endless checklist that makes every later investigation prohibitively slow.
Check the counterfactual: would the proposed improvement have been feasible with information available then, and could it have changed the conclusion? A rule requiring undisclosed private data may be impossible. The appropriate improvement might be to label the thesis untestable rather than estimate that missing variable confidently.
This review is not a search for blame and not a performance attribution system. Price outcomes, business outcomes and portfolio effects require different measurements. Keep them linked but distinct so a rising market does not hide a broken operating thesis or a falling market condemn sound evidence work.
A further check on the example
Add a timing audit to the postmortem. List when the weak order frequency first became observable, when the research note recognized it and when the thesis was revised. A long gap can reveal a monitoring failure even if the original model was reasonable. If the information arrived only after losses were reported, the main issue may instead be that the original claim depended on an unobservable variable. Different failures need different process remedies.
FAQ
Should successful theses also be reviewed?
Yes. They reveal whether the mechanism worked or whether an unrelated favorable event rescued the outcome. Reviewing only failures can miss lucky processes.
How do I avoid hindsight bias?
Preserve the original record, use publication dates and explicitly separate what was observed then from explanations learned later. Do not silently replace forecasts with actuals.