Study synthesis: separate a pooled mean from differing effects
A pooled mean can be positive even when the effects being summarized differ substantially. Before combining studies, define a compatible effect and its standard error, verify independence, and distinguish uncertainty about an average from variation across underlying effects. Counting favorable papers or averaging their headline numbers cannot answer those questions.
The output should retain every included estimate, its uncertainty, the pooling assumptions and a heterogeneity assessment. A synthesis is a model of comparable evidence, not a way to make incompatible or biased studies reliable by adding more rows.
Establish the common question before assigning weights
Suppose five fictional, independent task experiments compare the same evidence-checking procedure with the same control. Their outcome is the percentage-point change in correctly verified claims under a compatible assessment rule. Each estimate has standard error one percentage point. The numbers below teach synthesis arithmetic; they are not measured product results or an investment-return forecast.
The Cochrane Handbook chapter 10, particularly sections 10.3 and 10.10, discusses inverse-variance methods and heterogeneity. Its evidence-synthesis guidance does not certify this hypothetical dataset. In actual research, first retrieve the comparison, outcome definition, sample, analysis and reported uncertainty for each study.
Record the effect direction and scale. A percentage-point difference is not an odds ratio, a percentage change or a standardized score. Correct compatible sign conventions explicitly. If the reported standard error concerns another contrast or ignores the original clustering, do not silently reuse it as an eligible weight.
Rebuild a common-effect calculation
Assume the five estimates are zero, two, four, six and eight percentage points. Under a common-effect inverse-variance calculation, each weight is one divided by its squared standard error, so all weights equal one.
| Study | Effect, percentage points | Standard error | Weight | Weighted effect |
|---|---|---|---|---|
| A | 0 | 1 | 1 | 0 |
| B | 2 | 1 | 1 | 2 |
| C | 4 | 1 | 1 | 4 |
| D | 6 | 1 | 1 | 6 |
| E | 8 | 1 | 1 | 8 |
| Total | — | — | 5 | 20 |
The pooled estimate is twenty divided by five, or four. Under the stated independent, compatible variance inputs, its standard error is the square root of one-fifth, approximately 0.4472 percentage points. The arithmetic is simple because the within-study uncertainties are equal, not because equal weighting is generally appropriate.
This calculation does not establish that the common-effect assumption fits the situation. It describes what the selected model would produce. Choose the scientific target before seeing which model gives a narrower interval or a more favorable conclusion.
Quantify the dispersion rather than counting votes
Compute the weighted squared deviations from four. They are sixteen, four, zero, four and sixteen, giving Cochran’s Q of forty with four degrees of freedom for this comparison. The spread deserves investigation; Q is not an effect size and does not identify the reason the studies differ.
For a transparent teaching calculation, use the DerSimonian–Laird moment rule: estimated between-study variance equals the positive part of Q minus k-minus-one, divided by the sum of weights minus the sum of squared weights divided by their sum. Here the denominator is five minus five-fifths, or four. The variance estimate is (40 − 4) / 4 = 9 squared percentage points.
tau_squared = max(0, (Q - (k - 1)) / (sum_w - sum_w_squared / sum_w))
sum_w_squared means the sum of each individual squared weight.Its square root, three percentage points, is on the effect scale. Under this chosen moment calculation, the corresponding Q-based I-squared is 90%. That is not a claim that 90% of studies are wrong, that the effect is ninety percentage points or that every population has the same uncertainty.
The implementation discussion in section 10.10.4.4 explains that other variance estimators can perform better. This hand calculation is not a universal method recommendation. With only five studies, estimating dispersion remains difficult even when the point calculation is reproducible.
Add between-study variance to the weights
A random-effects inverse-variance calculation uses one divided by within-study variance plus estimated between-study variance. Each new weight is therefore 1 / (1 + 9) = 0.1. The weights sum to 0.5; the weighted estimates sum to two; the pooled mean remains four.
The mean staying unchanged does not make heterogeneity irrelevant. The simple model-based variance of the mean becomes one divided by 0.5, or two, and its standard error becomes approximately 1.4142. Equal within-study variances make the two means coincide in this special example. With unequal uncertainties, the weighting and pooled mean can both change.
Do not describe random effects as a general correction for weak studies. The model introduces a distribution of related effects. It does not correct selective reporting, incompatible measurements or duplicated participants. Large estimated dispersion can also give smaller studies relatively more influence than they had in a common-effect analysis.
For a separate unequal-precision sensitivity, consider only two invented estimates: zero with standard error one, and eight with standard error four. Their common-effect weights are one and 0.0625, giving a mean of approximately 0.470588. If between-study variance is held at nine as an explicitly supplied scenario input, the random-effects weights become 0.1 and 0.04, giving approximately 2.285714.
The second report gains relative influence because adding nine reduces the proportional difference between the variances. This is not a claim that the two-study dataset estimated a variance of nine; the value was fixed for the sensitivity. In a real synthesis, report the chosen estimation method and resulting weights. The example shows why a changed pooling model can shift the mean when uncertainties differ, even though it did not shift the five-study mean above.
Distinguish an average interval from a new-effect interval
Keep the interval rule visible. For this example, the random-effects residual scale calculation is 0.1 × 40 / 4 = 1. A corresponding adjusted mean variance is still two. Using an illustrative t multiplier with four degrees of freedom, approximately 2.7764, gives an approximate nominal 95% mean interval of four plus or minus 2.7764 times square root of two: about 0.07 to 7.93 percentage points.
An approximate nominal 95% prediction interval for the underlying effect in a similar new study includes between-study variance as well. Using the same multiplier, four plus or minus 2.7764 times square root of eleven gives approximately −5.21 to 13.21. Its target differs from the mean interval, and it allows negative underlying effects despite a positive pooled mean.
The prediction-interval discussion in section 10.10.4.3 describes the target and cautions about distributional assumptions and few studies. These are approximate model-based intervals, not guaranteed coverage. Uncertainty in the variance estimate remains important. Predicting a future observed study estimate would additionally involve that study’s sampling error; do not label this interval as covering every future reported estimate.
Investigate a different boundary: incompatible targets
Suppose a sixth report measures time saved rather than correctly verified claims. Another evaluates a different procedure against an easier control. Adding their headline numbers to this table would not simply make the pooled evidence stronger. Their targets must first be reconciled, or kept in separate syntheses.
Similarly, if two reports reuse the same participants, their errors may be correlated. A diagonal weight table that assumes independence is no longer justified. Retrieve the dependence information or narrow the synthesis; do not invent a discount percentage merely to retain every favorable report.
Investigate differences in tasks, implementation, populations and risk of bias before offering a mechanism for dispersion. A moderator analysis with five studies has very limited support. Choosing subgroups because they remove an inconvenient result is another selection step, not a demonstrated explanation of heterogeneity.
Deliver the synthesis ledger
| Field | Evidence to retain | Check |
|---|---|---|
| Study identity | Report and underlying sample | Independent evidence or modeled dependence? |
| Effect target | Outcome, comparison, direction, units | Same interpretable quantity? |
| Uncertainty | Standard error and original design | Valid for the selected estimate? |
| Weighting | Common/random target and variance method | Reproducible weights and pooled mean? |
| Dispersion | Q, variance, assumptions | Variation separated from average precision? |
| Intervals | Target, multiplier and limitations | Mean, true new effect or observed estimate? |
Update the entire ledger when a study is added, a standard error is corrected or overlapping data are discovered. Recalculate heterogeneity and all affected weights rather than appending a new estimate to an old pooled average. The conclusion should identify which average is supported, how effects differ and where the evidence is too limited for a confident new-setting claim.
FAQ
Does the unchanged pooled mean make random effects unnecessary?
No. Equal weights can leave the mean unchanged while dispersion and uncertainty change substantially.
Can random effects repair incompatible outcomes or reporting bias?
No. Those require separate evidence assessment and an appropriately defined synthesis.