Probability claims: test calibration with outcomes and scores
A probability claim is calibrated when events assigned similar probabilities occur at corresponding frequencies, within the limits of the available evidence. Test those frequencies on outcomes that have matured, and report an appropriate scoring rule alongside them. A model’s confident wording, good ranking or high classification accuracy does not independently establish calibration.
Define a probability that can be checked
Suppose a research assistant assigns probabilities that a document excerpt contains a particular disclosure. The event must have a written definition and a reference-label process. A probability of finding that disclosure is not a probability that the issuer will outperform or that the extracted accounting judgment is legally correct. Keep the event narrower than the conclusions the evidence can support.
The scikit-learn probability calibration guide explains reliability diagrams and distinguishes calibration from the broader properties assessed by proper scoring rules. In particular, Brier loss reflects more than calibration alone. The worksheet here evaluates hypothetical forecasts rather than benchmarking an existing CoTrading feature.
A hypothetical four-case calculation
Imagine four locked predictions, each assessed against a later independent reference label. Predicted probabilities are 0.8, 0.8, 0.2 and 0.2. Outcomes are 1, 0, 1 and 0. The mean squared probability error, using the ordinary binary Brier convention, is 0.34.
| Case | Predicted probability | Reference outcome | Squared error |
|---|---|---|---|
| A | 0.8 | 1 | 0.04 |
| B | 0.8 | 0 | 0.64 |
| C | 0.2 | 1 | 0.64 |
| D | 0.2 | 0 | 0.04 |
The sum is 1.36, divided by four. A constant forecast of 0.5 would score 0.25 on these same cases. This tiny example shows the arithmetic, not a reliable estimate of deployment quality. With two observations per probability group, sampling uncertainty is substantial, and one changed label can move the apparent result sharply.
Build a reliability table
For a larger untouched evaluation set, group forecasts using a declared binning rule. Report the mean predicted probability, observed positive proportion and number of cases in each bin. In the toy example, both groups have an observed frequency of 0.5 despite predictions of 0.8 and 0.2. Do not hide the small group sizes behind a smooth calibration curve.
Bin width and sample composition affect the picture. Large bins can conceal errors within a group; tiny bins can display noisy fluctuations as meaningful structure. Keep a transparent table and examine a few sensible bin definitions. A visual curve is an estimate, not proof that every individual probability has the stated interpretation.
Separate calibration, resolution and decisions
A constant base-rate forecast can be calibrated in the aggregate while failing to distinguish easy cases from difficult ones. A good ranking can also coexist with overstated probabilities. Compare discrimination and calibration separately, then discuss what the review threshold is intended to achieve. One favorable metric should not stand in for every property of a research workflow.
The Brier score rewards accurate probability forecasts under its scoring convention, but a lower score alone does not show which component improved. Document any scaling convention, sample weighting and treatment of missing outcomes. Do not compare scores calculated on different label definitions or case populations as though they measured one stable task.
Keep recalibration out of its own examination
If you fit a correction using labeled examples, reserve independent cases to evaluate the corrected probabilities. Choosing bins, correction methods and thresholds repeatedly on the final test set turns it into development data. Record which cases were used for fitting and which remained untouched.
Also track outcome maturity and label disagreement. Pending cases cannot simply become negatives, and adjudicated labels can still contain errors. Recheck calibration when languages, document types or prevalence change. A probability assessment is a dated empirical statement about a defined task and population, not a permanent certificate of model reliability.
FAQ
Does an 80% probability mean the individual case will succeed?
No. Calibration concerns frequencies among comparable predictions. An individual binary outcome is still either observed or not observed under the label rule.
Is lower Brier loss proof of better calibration?
Not by itself. It evaluates multiple aspects of probability quality; inspect reliability information and case composition as well.