Classifier alerts: recalculate precision at the target base rate
The precision of a research alert depends on how common the target problem is in the population being reviewed. Sensitivity and specificity do not determine precision on their own. Before deploying a filing classifier, estimate the relevant base rate, hold the threshold definition constant and translate the resulting confusion matrix into an actual human review workload.
Define the positive class and the population
Suppose a tool flags whether a disclosure contains a contract change requiring specialist review. Positive means that a reviewer applying a written rule confirms that condition. It does not mean the company is a bad investment, the filing contains fraud or the share price will fall. Those are different targets requiring different labels and evidence.
The scikit-learn confusion-matrix documentation distinguishes true and false classifications against reference labels. Use that structure to keep denominators explicit. The deployment calculation here is an original hypothetical worksheet, not measured performance of an available CoTrading classifier.
A hypothetical base-rate change
Assume a fixed threshold has sensitivity of 80% and specificity of 95%. In one population, 10% of disclosures contain the target condition; in another, only 1% do. Imagine 10,000 disclosures in each population and assume, solely for this illustration, that the two conditional error rates remain unchanged.
| Quantity | Base rate 10% | Base rate 1% |
|---|---|---|
| Truly positive disclosures | 1,000 | 100 |
| True alerts | 800 | 80 |
| False alerts | 450 | 495 |
| Total alerts to review | 1,250 | 575 |
| Alert precision | 64% | About 13.9% |
In the low-prevalence population, false alerts outnumber true alerts even though specificity remains 95%. Precision is 80 / 575, not 80 / 100. The latter fraction is sensitivity: how many actual positives were caught. Mixing those denominators can make a costly review queue sound much more selective than it is.
Reconstruct the calculation
For prevalence p, sensitivity s and specificity c, expected positive predictive value is s × p divided by s × p + (1 - c) × (1 - p). Expected counts multiply each component by the population size. These are conditional projections under the stated assumptions; a prevalence estimate and error-rate estimates carry uncertainty that the simple point calculation does not remove.
Check how the evaluation set was collected. A collection deliberately balanced between easy positives and negatives may be useful for development, yet its raw precision does not represent a live filing stream. Keep the sampling rule and class proportions with the evaluation report. Do not infer deployment prevalence from the share of alerts without separately estimating the classifier’s errors.
Treat stable error rates as an assumption
Changing company size, language, document format or disclosure practice can change sensitivity and specificity as well as prevalence. Reweighting by a new base rate cannot repair such conditional performance drift. Audit representative documents from the new population, including unflagged cases, and distinguish observed evidence from the rates merely carried over from the earlier test.
Maintain the same target definition and threshold when comparing populations. Raising the threshold may reduce false alerts but also miss genuine conditions. Evaluate that tradeoff against the consequence of an omission and the capacity of the review team; do not describe improved precision alone as an unqualified improvement.
Turn rates into review operations
State the expected number of alerts, likely confirmed cases and unresolved false-negative burden for a defined period. In the second example, 575 reviews produce an expected 80 true alerts under the assumptions, while 20 positives are missed. The number of documents available for review, not just an attractive percentage, determines whether the process is usable.
When the base rate is poorly known, show a plausible range and explain its evidence. Preserve the reference-label process and maturity date. An alert remains a request for examination rather than a verified company conclusion, regardless of its estimated predictive value.
FAQ
Is 95% specificity the same as 95% precision?
No. Specificity concerns actual negatives; precision concerns returned positives. Their denominators and business interpretations differ.
Can I reuse last year’s precision directly?
Only with evidence that the relevant population, target and operating characteristics remain suitable. Recheck prevalence and conditional errors instead of carrying one headline score forward.