Evaluate broker forecasts at a consistent forecast horizon

CoTrading editorial team 2026-10-27 阅读中文全文

To evaluate a broker’s earnings forecast, compare an archived prediction with a compatible actual result at a declared lead time. A forecast made 30 days before results has a different information set from one made 180 days before results. Put them in separate groups before calculating error. The deliverable is a forecast-to-actual panel with selection dates, missing observations and accuracy measures, rather than a leaderboard assembled from whichever predictions are easiest to find today.

The example uses fictional annual diluted EPS for one company and two brokers. Historical snapshots are externally supplied inputs. The script constructs synthetic archive fixtures for demonstration; it cannot reconstruct authentic old forecasts from a current report. Neither Vibe nor CoTrading is assumed to supply a historical consensus archive, broker coverage history or licensed research documents.

Obtain the history that the question requires

For every prediction, collect company identifier, broker, target fiscal year, EPS basis, currency, forecast value, original source identifier and the timestamp when that version was frozen. Retain the corresponding report when permitted. A spreadsheet containing an old target year but downloaded today is insufficient evidence of what a broker predicted before a past earnings release. The timestamp needs to describe the archived forecast version, not the date someone later exported the table.

Collect actual EPS with its fiscal period, release date and source version. Decide whether evaluation uses the first reported actual or a specified restatement. Preserve that choice consistently. IAS 33’s official overview distinguishes basic and diluted EPS. Here both sides are diluted, CNY per share, for the same annual period. Do not treat an adjusted broker estimate as comparable to statutory EPS without an explicit reconciliation.

Our actuals are FY2024 EPS of 1.00, released on 2025-03-01, and FY2025 EPS of 1.20, released on 2026-03-01. These dates describe fictional fixtures, not a real issuer’s calendar. Keep the actuals outside the forecast selection step. Selection should depend on the frozen date and scope, not on which candidate prediction later turned out closest to the answer.

Define equal lead times before looking at errors

We evaluate 180-day and 30-day horizons relative to the formal actual release. For each company-year-horizon, subtract the horizon from the release date to produce the cutoff. Select the latest archived forecast no later than that cutoff, allowing at most three calendar days of staleness. This tolerance is a declared rule for the example. A production study may use a different policy, but must keep it fixed across brokers within the comparison.

Python datetime provides date arithmetic used for the cutoff. This fixture is daily: there is no intraday selection and no claim that a calendar-day tolerance equals a trading-day tolerance. With timestamps, define the timezone and what happens when reports are released after market close. Missing archive data should remain missing; it should not be repaired with a later forecast merely because the target fiscal year matches.

If two archived records for the same broker, target year and selected day contain different EPS values, stop that observation as a conflict. Without publication order evidence, choosing the first or last row makes the result depend on input order. Equal values may be deduplicated with source references retained. The complete example checks that reversing conflicting rows produces the same rejection.

Calculate error with an explicit sign convention

Define error as forecast minus actual. A positive error is an overestimate and a negative error is an underestimate. Mean absolute error, MAE, averages the absolute differences in CNY per share. Bias averages the signed errors. Report the number of evaluated company-years and the number eligible, because small samples can appear accurate simply by excluding difficult periods.

Forecasting: Principles and Practice explains MAE and problems with percentage errors near zero. Our script reports a relative error only when actual EPS is above 0.05. This is a deliberately conservative fixture rule, not a universal financial reporting threshold. Nonpositive and near-zero actuals retain their absolute difference and an unavailable relative measure. NaN and either infinity are rejected as invalid observations rather than being called a large forecasting error.

f, a = finite(forecast), finite(actual)
difference = f - a
relative = difference / a if a > D("0.05") else None

The downloadable implementation uses decimal arithmetic as well as these eligibility checks. Keep unrounded error values for aggregation, then format the displayed table. Rounding each observation before calculating an average can change a small comparison and makes reconciliation harder.

Read the two-horizon worked example

At 180 days, broker A forecasts 1.20 for FY2024 and 1.00 for FY2025. Against actuals of 1.00 and 1.20, its errors are plus 0.20 and minus 0.20. MAE is 0.20 and bias is zero, with coverage of two out of two. Zero bias does not mean accurate: opposite errors cancel in the signed average while absolute error remains visible.

Broker B’s 180-day archive contains only FY2024 at 0.90. Its error is minus 0.10, MAE is 0.10 and coverage is one out of two. B’s FY2025 forecast of 1.18 exists only at 30 days. The selector does not borrow it for the 180-day panel. It would be misleading to declare B superior on the basis of its smaller MAE without acknowledging both the missing period and the tiny sample.

At 30 days, A has 1.05 and 1.15, so its MAE is 0.05 over two observations. B has 0.98 and 1.18, giving MAE 0.02 over two observations and bias minus 0.02. These are separate short-horizon results. Do not combine the two horizons into an overall average: doing so would confound forecast skill with how close each prediction was made to the earnings release.

Download the complete example. Use Python 3.12+, the standard library and synthetic data; run python R547.py. The printed output includes both horizon panels, coverage counts, missing years and a near-zero counterexample. Assertions test the selection rule, arithmetic, conflicting dates and nonfinite values on both forecast and actual sides.

Keep coverage and unstable denominators visible

A different failure mechanism appears when actual EPS is 0.01 and the forecast is 0.03. The absolute error is 0.02 CNY per share, but the relative figure is suppressed under the declared near-zero rule. Substituting a denominator of one, dropping the observation entirely or printing an impressive percentage would each change the study’s meaning. Retain the observation, the absolute difference and the reason the relative measure is unavailable.

For a larger study, add a common-sample comparison using only company-years covered by every included broker, alongside each broker’s full eligible panel. Do not silently replace the latter with the former. Show sector, company and horizon composition before pooling scales. MAE in EPS units is directly readable within the same company; comparisons across companies with different share denominations require an additional carefully justified normalization.

Deliver a panel that can be updated without rewriting history

Save a row for each expected broker-year-horizon, including missing rows. Record cutoff, selected frozen date, source identifier, actual version, error, relative eligibility and rejection reason. Aggregates should be reproducible from accepted rows, and totals should reconcile to the expected panel size. Keep conflicted and missing records out of the numerator while retaining them in coverage diagnostics.

When a new year matures, append its archived forecasts and compatible actual. When an actual is restated or an archive error is corrected, create a new evaluation version and show the affected rows. A current report may help future forecasts, but cannot fill an absent past snapshot. The final conclusion should describe the measured sample and uncertainty; this small example supports an evaluation method, not a recommendation to trust a particular broker.

Keep the forecast selection table as a separate artifact from the scoring table. A reviewer can then check historical eligibility without seeing errors, reducing the temptation to favor versions that happened to predict the final result well.

FAQ

Can I use the most recent forecast for each past fiscal year?

Only if the research question explicitly concerns that timing. Equal-lead evaluation requires versions actually frozen by the historical cutoff.

Does zero bias establish good forecasting?

No. Positive and negative errors can cancel. Read bias together with MAE, coverage and the individual observations.