Funnel asymmetry: investigate missing studies without counting them
A funnel plot can reveal that less precise studies tend to report different effects. It cannot, by its shape alone, tell you how many studies are absent or why they are absent. Investigate reporting decisions through study plans, status records and available results before interpreting an empty part of the plot as proof of suppressed negative evidence.
The deliverable is a study-level reporting ledger linked to the effect-and-precision table. It separates studies missing from a search, completed studies without a report and reports that omit a planned outcome. Those mechanisms require different evidence and should not be compressed into one missing-publication count.
Establish the studies behind the plotted points
Define the intended comparison, outcome, eligible study designs and search cutoff. Assign one study identity to related registrations, protocols and reports, retaining their links. Several articles about the same experiment are not several independent plotted studies.
The Cochrane Handbook chapter 13 discusses missing evidence and the alternative explanations for funnel asymmetry. Locate sections 13.3.4.2 through 13.3.4.4 for plot interpretation and testing limitations. Its methodological discussion motivates the following invented research-workflow example; no actual publications are accused of selective reporting.
For every included estimate, preserve the outcome scale, effect direction, standard error and analysis that produced it. Choose the vertical-axis convention explicitly: standard error is larger for less precise estimates, whereas inverse standard error runs in the opposite numerical direction. A plot label must tell readers which is shown.
Rebuild a hypothetical asymmetric table
Suppose eight fictional experiments compare a verification aid with a common control. Effects are percentage-point differences in correct claim checks, with positive values favorable to the aid. Their standard errors and target definitions are assumed compatible for this illustration.
| Experiment | Effect, percentage points | Standard error | Observed pattern |
|---|---|---|---|
| A | −0.1 | 0.1 | More precise |
| B | 0.0 | 0.1 | More precise |
| C | +0.1 | 0.1 | More precise |
| D | 0.0 | 0.1 | More precise |
| E | +0.8 | 0.5 | Less precise |
| F | +1.0 | 0.5 | Less precise |
| G | +1.2 | 0.5 | Less precise |
| H | +1.4 | 0.5 | Less precise |
The less precise points are all positive while the more precise points cluster near zero. That is a description of these eight observed results. It does not establish an expected total study count. There is no valid subtraction that turns an apparently empty left-hand region into a known number of unpublished experiments.
Keep the unrounded numerical table and original uncertainty calculations. If a standard error was reconstructed, retain its assumptions. A malformed interval, reversed sign or duplicated report can create a visual problem before any reporting-bias mechanism needs to be invoked.
Compare two mechanisms producing the same appearance
One possible story is selective availability: small experiments with favorable results become accessible more often, while others do not. To investigate it, seek records showing experiments were planned or completed and whether availability depended on results. The observed points alone do not provide those records.
A different story is real task heterogeneity. Suppose E through H deliberately study difficult, fragmented documents where the aid has more opportunity to help, while A through D study simple documents. Then size or precision is associated with the underlying task effect. The same table could appear even if every completed experiment were reported.
These explanations can also coexist. Record document difficulty, procedure version, comparator and assessment rule alongside precision. Ask whether the effect difference remains interpretable within comparable task groups, while recognizing that four studies per group offer limited information. Do not declare the heterogeneity explanation proven merely because a convenient label can be attached after seeing the plot.
The chapter’s alternative-causes table also notes study limitations, chance and mathematical relationships between some effect measures and their standard errors. A generic plot is therefore an investigation trigger rather than a diagnosis. In particular, an analysis appropriate for one effect measure need not be appropriate for a standardized mean difference or an odds ratio.
Use planned-study records to identify known gaps
Now suppose a separately maintained, timestamped register identifies twelve fictional study plans. Eight are A through H above. The other four have the following documented status at the search cutoff.
| Plan | Status evidence | Target result availability | Current classification |
|---|---|---|---|
| I | Completed | No report located | Whole-report gap to investigate |
| J | Completed; report available | Planned target outcome absent | Within-study result gap |
| K | Still collecting data | Result not yet expected | Ongoing at cutoff |
| L | Cancelled before data collection | No experiment performed | Pre-collection cancellation |
The ledger contains twelve plans and eight currently available target estimates. There are two known completed-study gaps, with different reporting mechanisms. It would be wrong to call all four other plans suppressed negative studies. Neither I nor J’s missing target result has a known sign in this example.
Also distinguish number of reports from number of target estimates: J has a report, but it does not supply the required result. A search returning that report has not solved the outcome-availability problem. Conversely, an experiment omitted by the search but already fully reported is a retrieval failure, not necessarily selective non-reporting.
Among the ten documented completed studies, nine have a located report but only eight have the target result. Those are 90% report availability and 80% target-result availability for this known completed set. Dividing eight by all twelve plans answers a different question because the denominator includes ongoing and never-started experiments. None of these percentages measures coverage of unknown studies outside the register.
Assess the register’s scope too. If registration was voluntary or used by only some teams, a complete search of it cannot establish a complete experiment inventory. Record the participating groups and time window. Lack of a registration is not evidence that an experiment never occurred.
Check the protocol version that existed before results were known. A later amended document can legitimately explain a changed outcome, or can leave a reporting concern unresolved. Save both versions and their dates instead of treating the current webpage as proof of the original plan.
Keep testing limitations and sensitivity assumptions explicit
With only eight studies, a formal asymmetry test can have very limited power. The Cochrane testing discussion gives at least ten studies as a rule of thumb and describes other restrictions. Ten is not a certificate that any chosen test is valid, and a nonsignificant result does not demonstrate complete reporting.
This example therefore makes no asymmetry p-value claim. It retains the descriptive pattern and the documented gaps. When a formal test is appropriate, record the effect measure, uncertainty inputs and test assumptions; do not select whichever test makes the reporting concern disappear.
A sensitivity analysis may ask what conclusions would change under stated missing-result scenarios. Those scenarios must remain assumptions. Placing artificial negative points into a graph cannot turn them into located studies, and a model-adjusted count is not a verified inventory of hidden experiments.
If I’s result later becomes available, replace its unknown status with the actual estimate and source, preserving the earlier ledger. Do not keep a convenient imputed result because it supports the original suspicion. New evidence can either strengthen or reduce concern about selective reporting.
Deliver the reporting-investigation record
| Field | Evidence to save | Question answered |
|---|---|---|
| Study identity | Protocol, registration and report links | One experiment or repeated reports? |
| Eligibility | Comparison, outcome and cutoff | Does it belong in this synthesis? |
| Effect/precision | Estimate, standard error, method | Is the plotted point reproducible? |
| Status | Planned, ongoing, completed, cancelled | Should a result exist by the cutoff? |
| Availability | Whole report and target outcome separately | Which evidence is actually missing? |
| Explanation | Design differences, reporting records, unknowns | What could distinguish the mechanisms? |
Revisit when registrations are updated, completed reports appear, outcome definitions change or overlooked reports are located. Preserve unresolved cases and avoid attributing motive without evidence. The useful conclusion identifies observed small-study differences and specific reporting gaps; it does not claim a visual pattern has counted every missing study or established why any particular investigator did not report a result.
FAQ
Does a symmetric funnel prove reporting is unbiased?
No. Some missing-evidence mechanisms leave little visible asymmetry, especially with few studies.
Can an asymmetric funnel tell me how many studies were suppressed?
No. A verified count needs independent study records; statistical adjustments remain model-dependent scenarios.