Counterfactual comparisons: test the comparison group
A comparison group is useful only if it can represent a defensible counterfactual for the question. Similar company names or industries do not automatically establish that condition. Before AI describes a difference as an effect, inspect how groups were selected, what changed before the event and which alternative mechanisms could explain the result.
The World Bank’s Impact Evaluation in Practice introduces causal inference, counterfactuals and comparison designs. The worksheet here is a research diagnostic, not a substitute for those designs. It deliberately separates a descriptive arithmetic contrast from a justified causal interpretation.
State the untreated question
Define the treated population, event, outcome and observation unit. Then write what would have happened to that same population without the event. That unobserved outcome is the target of the comparison argument. A group that is easy to download is not necessarily an adequate stand-in.
Record why the comparison entities did not receive the event or treatment. Voluntary adoption, eligibility rules and location can create different selection mechanisms. If adoption depends on expected performance, a later gap can reflect selection as well as the event. Do not let AI treat group membership as randomly assigned without evidence.
Match population definitions and measurement. Census SUSB methodology shows that business statistics have explicit coverage rules. In a company study, inspect such scope differences before using an external population as a comparator. Compatible labels do not guarantee comparable units.
Hypothetical before-and-after contrast
Assume a fictional adopting group has an outcome index of 100 before an event and 120 after it. A non-adopting comparison group moves from 80 to 88. The adopter’s change is 20, the comparator’s is eight, and the difference in changes is 12 index points under this additive setup.
The 12 is arithmetic, not automatically the event’s causal effect. Suppose adopters were already growing faster in earlier periods or were selected because of a different market opportunity. Those mechanisms challenge the comparison. One pre-event observation cannot demonstrate that the groups would have followed the required comparable path.
Now suppose the event affects suppliers serving both groups. The comparator may also be exposed, changing what untreated means. Record that interference rather than assuming the non-adopting label guarantees no effect. All groups and values are hypothetical, and no causal estimate from real companies is claimed.
Comparison-group diagnostic
| Dimension | Question to document |
|---|---|
| Selection | Why did each group enter its category? |
| Pre-event path | Are several comparable prior observations available? |
| Measurement | Same outcome, units, period and reporting scope? |
| Common changes | Which market or policy shifts affect both groups? |
| Differential changes | What else changed for only one group? |
| Spillovers | Can the event affect the proposed comparator? |
Keep percentage changes separate from additive differences. In the example, starting levels differ, so a percentage-based calculation answers a different question. State the estimand and scale before selecting the formula. Do not switch scales after seeing which contrast looks strongest.
Report evidence against the comparison
Examine prior paths, placebo timings and alternative justified comparators as diagnostics, while acknowledging that passing them does not prove every identification assumption. If the data cannot address a critical selection mechanism, describe the result as an association or difference under assumptions.
Ask AI to list the strongest reasons the comparator may fail, supported by the available evidence. The final note should show the arithmetic, the counterfactual argument and its unresolved conditions separately. This makes the comparison useful for research without upgrading a convenient peer group into a causal control or predicting an investment outcome.
FAQ
Is a same-industry peer automatically a valid control?
No. Selection, earlier trends, exposure and measurement can differ within an industry.
Does subtracting the peer group’s change establish causality?
No. It calculates a contrast. Causal interpretation still requires a defensible design and the assumptions that connect the comparator to the unobserved outcome.