Factorial comparisons: test whether two changes work differently
Two workflow changes can each help in isolation without delivering the sum of those improvements when combined. A two-by-two factorial comparison makes that question explicit: measure neither change, A alone, B alone and both. Then compare A’s effect at each setting of B rather than declaring the best-looking cell proof that both components work independently.
The result should preserve four cell means, sample counts, the outcome scale, conditional effects and the prespecified interaction contrast. It also needs the assignment and repetition structure. Four attractive averages can describe a sample without establishing a reproducible causal mechanism or a statistically precise interaction.
Define the experiment before reading its result
Imagine a research team testing a document-retrieval aid, A, and an evidence-checklist aid, B. The outcome is completion time in minutes for a defined verification task, measured only alongside a separately specified accuracy requirement. These are hypothetical aids, not claims about available CoTrading features.
Specify eligible tasks, document versions, operator experience, timeout handling and the accuracy rule before assignment. Randomly assign suitable independent task sessions across the four combinations. If the same operator or issuer contributes repeated sessions, retain those identifiers and account for dependence. Randomizing operators would make the operator, not every subsequent click, the primary assignment unit.
The NIST interaction-effects case study illustrates checking interactions alongside main effects. Its engineering findings do not establish the effect of these invented research aids. We use a separate four-cell example to show the calculation and its limits.
Rebuild all four cells
Assume equal planned allocation and compatible measurements produce these fictional mean times. Lower time is favorable only if the required evidence and accuracy remain adequate.
| B setting / A setting | A off | A on |
|---|---|---|
| B off | 20 minutes | 16 minutes |
| B on | 18 minutes | 17 minutes |
A’s conditional effect with B off is 16 − 20 = −4 minutes. With B on it is 17 − 18 = −1 minute. Using the stated subtraction order, the additive interaction contrast is −1 − (−4) = +3 minutes. Equivalently, calculate 17 − 18 − 16 + 20.
The positive sign means A’s time reduction is smaller when B is on, on this minutes scale. It does not mean the combined workflow takes longer than the baseline: seventeen is still less than twenty. Nor does it establish that either aid is intrinsically harmful. The interaction concerns how one effect changes across the other factor’s settings.
If the isolated reductions added without interaction, the predicted combined time would be twenty minus four minus two, or fourteen. The observed seventeen is three minutes above that additive prediction. B’s conditional effects tell the same story: −2 minutes with A off and +1 minute with A on, a difference of +3.
Separate average main effects from conditional effects
With equal weighting of B’s two settings, A’s average main effect is (−4 − 1) / 2 = −2.5 minutes. B’s average main effect is (−2 + 1) / 2 = −0.5 minutes. The grand mean is 17.75 minutes. These averages summarize this particular weighting across the other factor.
Do not use the −0.5 average to conceal that B is slower by one minute when A is already on. A deployment decision involving only A-on sessions needs that conditional comparison. Conversely, a factorial average remains meaningful when its planned distribution is the target; interaction does not automatically make every main effect useless.
If cell sizes are unequal, state whether the summary uses equal cell weights or observed workload weights. A pooled difference can otherwise change its target with allocation or missing sessions. Preserve all four counts, means and relevant variance estimates rather than letting a software default select the scientific question silently.
Explain why software coefficients can differ
Under zero/one coding, a saturated mean model for this table is twenty minus four A minus two B plus three A×B. Its interaction coefficient is the difference of differences, three. This is a representation of the four means, not evidence that the model has estimated uncertainty from four numbers.
Under minus-one/plus-one coding, the same means have coefficients 17.75, −1.25, −0.25 and +0.75 for intercept, A, B and A×B. Substituting each setting reconstructs the table. Four times the interaction coefficient equals the difference of differences, while twice it equals the conventional interaction effect from comparing the product column’s two levels.
The NIST interaction-matrix explanation, especially its coding and model discussion, distinguishes the coded coefficient from the effect estimate. A report of 0.75, 1.5 or 3 can therefore use different conventions. Ask for the design coding and contrast definition before treating the values as contradictory.
Test a different boundary: additive and multiplicative scales
Consider another fictional table with times 100 for neither aid, eighty for A alone, fifty for B alone and forty for both. A multiplies time by 0.8 whether B is off or on. On a ratio scale there is no change in A’s relative effect, and the corresponding log-scale interaction is zero.
On the raw minutes scale, A changes time by minus twenty when B is off and minus ten when B is on. The additive interaction is therefore plus ten minutes. Neither calculation disproves the other. They ask whether absolute differences or relative changes combine without interaction.
Choose the scale according to the decision and measurement properties before examining which gives the preferred story. A positive interaction on one scale is not a universal label for biological, cognitive or commercial synergy. Also keep an accuracy outcome separate: a fast workflow that omits required evidence may fail the task regardless of its timing contrast.
Keep precision and identification visible
One observation in each cell fits four cell means exactly and supplies no within-cell estimate of pure error. Repeated, appropriately independent sessions are needed to assess variation. Record the planned interaction contrast and use uncertainty calculations consistent with randomization, pairing, clustering and the outcome distribution.
For a separate precision illustration, suppose each cell contains five independent sessions with sample standard deviation two minutes. Each mean has estimated variance four divided by five. Under independent cell means, the variance of the four-term interaction contrast is four times that amount, or 3.2; its estimated standard error is about 1.79 minutes. The point contrast remains three, but its uncertainty is substantial. A confidence interval needs the appropriate distribution and degrees of freedom, not an automatically attached normal cutoff.
A significant effect for A with B off and a nonsignificant effect with B on does not by itself establish a significant difference between those effects. Evaluate the interaction contrast directly. Conversely, an imprecise interaction estimate does not prove exact additivity; report its interval and the practically important differences it still allows.
Random assignment supports a causal interpretation only with the relevant execution and analysis assumptions. If difficult tasks systematically receive both aids, the four observed averages alone do not separate interaction from task selection. Preserve exclusions, failures and missing outcomes instead of keeping only successful fast sessions.
If one cell’s mean includes timeouts while another includes only completed tasks, first repair the outcome definition. Report completion counts and the prespecified handling of failures; a shorter successful-case mean can coexist with worse overall task completion.
Deliver a reproducible factorial record
| Record | Required content | Review question |
|---|---|---|
| Factor definitions | A/B off and on, versions | Is each combination actually comparable? |
| Assignment | Unit, allocation and dependence | Does inference match randomization? |
| Outcomes | Time, accuracy, failures, counts | Were slow or failed tasks omitted? |
| Four cells | Means and variation | Can every contrast be reconstructed? |
| Interaction | Scale, subtraction order, interval | Is it a direct comparison of effects? |
| Decision | Target workload and acceptable tradeoff | Which conditional effect is relevant? |
Reopen the comparison when the checklist changes, the retrieval aid is updated or the task mix shifts. Retain the original factor definitions and rerun the intended combinations on suitable new tasks. The useful conclusion identifies how the components behave together under stated conditions; it does not turn one favorable combination into a promise that independent benefits will always add up.
FAQ
Does the fastest combined cell prove both components help?
No. Compare each component with its counterpart at the relevant setting and retain accuracy and uncertainty.
Is statistical interaction the same on every scale?
No. Absolute differences, ratios and logarithms can yield different interaction statements because their targets differ.