Randomization tests: permute assignments allowed by the design
A randomization test asks how unusual the observed statistic would be under the study’s actual assignment mechanism and a stated null hypothesis. It does not authorize every possible shuffle of the data. Pair restrictions, cluster assignment and unequal assignment probabilities are part of the reference distribution, not optional adjustments after a p value is calculated.
The deliverable is a list or reproducible generator of legal assignments, the statistic for each assignment, and a tail calculation that includes the observed assignment. For a small design, enumerate the entire space so another reviewer can see exactly what is being counted.
Start with the assignment record and the null
Assume a hypothetical experiment randomly assigns two of four documents to a new research workflow, with the other two using the established workflow. Each of the six two-document subsets has equal probability. Document identities are A, B, C and D; their observed numerical outcomes are 1, 2, 3 and 4 respectively.
Use the sharp null that the workflow changes no document’s outcome, with well-defined treatment versions, no interference between documents and complete observed outcomes. Under that null, each document would retain its recorded outcome under every allowed assignment. This lets us calculate statistics for assignments that were not actually implemented.
The sharp null is stronger than “the average effect is zero.” Individual positive and negative effects can average to zero without revealing every missing potential outcome. Holding all outcomes fixed and calling the resulting procedure an exact test of any zero-average-effect model would overstate what has been justified.
World Bank DIME’s design appendix, in its randomized-trial and matching discussions, emphasizes preserving how treatment and control groups were constructed. The numerical design here is original. A shuffled observational dataset does not inherit the experiment’s assignment evidence.
Enumerate six legal assignments
Define the statistic as new-workflow mean minus established-workflow mean. Suppose the observed treated set is A,B, giving (1+2)/2−(3+4)/2=−2. Under the sharp null, enumerate:
| Assigned new workflow | Assigned established | Mean difference |
|---|---|---|
| A,B | C,D | −2 |
| A,C | B,D | −1 |
| A,D | B,C | 0 |
| B,C | A,D | 0 |
| B,D | A,C | 1 |
| C,D | A,B | 2 |
Each row has probability 1/6. The two zero statistics are distinct assignments and must both be retained. Counting only unique statistic values would replace six equally probable assignments with five incorrectly weighted values.
For the prespecified lower-tail alternative, the exact p value is the probability of T ≤ −2, equal to 1/6. The observed A,B assignment is that one row. Removing it and looking only for a more extreme result among the other five would incorrectly give zero.
For a prespecified absolute-value two-sided rule, count |T| ≥ 2: two rows give 2/6=1/3. State this rule before interpreting the result. Two-sided conventions are not universally identical; in this symmetric example the absolute tail agrees with twice the smaller one-sided tail.
Match software settings to the design
SciPy’s permutation_test documentation, at permutation types, exact enumeration and alternative definitions, distinguishes independent reassignment, within-pair sample swaps and random pairing. Its default two-sided convention doubles the smaller one-sided p value. A generic function call is therefore not a substitute for specifying the intended legal assignments and tail.
For the six-row design, exact arithmetic is sufficient. With floating-point statistics, theoretical ties may differ by tiny numerical errors. Inspect the null distribution and use a justified equality tolerance or an exact statistic where feasible. Otherwise a value mathematically equal to the observed statistic can be wrongly excluded from its tail.
The software documentation consulted is the v1.18.0 manual. The table is independently enumerated and does not depend on claiming a particular package version generated the experiment. Save the implementation version when software is actually used so defaults and tie handling remain reproducible.
A paired design has a different assignment space
Now consider a separate hypothetical protocol that forms pairs A,B and C,D and assigns exactly one document in each pair to the new workflow. The legal treated sets are only A,C; A,D; B,C; and B,D. Their statistics are −1, 0, 0, 1. A,B is illegal because it treats both members of the first pair and neither of the second.
Suppose the observed paired assignment is A,C, with T=−1. Its lower-tail p is 1/4 and its absolute two-sided p is 2/4=1/2. Under an unrestricted two-of-four reference space, that same observed T would instead have absolute-tail probability 4/6=2/3. These are different assignment mechanisms; choose the one that actually generated the experiment, not the one producing the preferred p value.
This is a new protocol branch with a legal observed assignment. It is not a retrospective reclassification of the original observed A,B assignment as paired. An assignment log that contradicts the claimed pairing needs investigation before a test is reported.
If teams rather than documents were randomized, shuffle teams and preserve their member documents together. For stratified assignment, preserve the allocated counts within each stratum. If assignments have unequal known probabilities, sum the probabilities of extreme assignments instead of using an unweighted row fraction.
Save the allocation log’s export date and version. A later label correction should not overwrite the original record without explaining whether it fixes transcription or revises the claimed protocol; those changes have different implications for the legal assignment space.
Separate exact enumeration from Monte Carlo approximation
Large legal spaces may require random draws from the stated assignment mechanism. Record the number of draws, generator state, statistic, tail rule and treatment of the observed assignment. A conventional conservative randomized-tail calculation is (b+1)/(B+1), where b sampled statistics are at least as extreme among B draws.
For an independent large-design illustration, B=999 and b=0 give 0.001, not zero. This is not a simulated result from the enumerable four-document example. It explains the correction used when a reference distribution is sampled rather than exhaustively enumerated. Repeating the Monte Carlo run adds simulation variability; it does not add experimental documents.
Do not add one mechanically to an already complete exact enumeration. Its observed assignment is already present in the full denominator. Similarly, sampling outcome rows with replacement is a bootstrap operation, not the legal randomization distribution of a fixed two-of-four experiment.
Audit the assignment test before reporting it
| Check | Evidence to retain | Response to a conflict |
|---|---|---|
| Recover design | allocation log, blocks, clusters and probabilities | do not invent exchangeability |
| Specify null | sharp outcome invariance and interference assumptions | distinguish it from a mean-only null |
| Define statistic | direction, covariate handling and analysis unit | keep the same rule in every reassignment |
| Generate assignments | legal sets or generator tied to the protocol | exclude impossible allocations |
| Count tails | observed assignment, ties and sidedness | verify counts independently |
| Report approximation | exact versus sampled, B and seed | retain Monte Carlo uncertainty |
Nonadherence also requires care. An assignment-based test and a test formed by shuffling actual treatment receipt answer different questions. Do not replace randomized labels with chosen receipt status and still call the test design-exact. Missing outcomes, selective exclusions or shared learning between workflows can likewise break the simple fixed-outcome argument.
A small p value is evidence against the specified null under the design; it is not the probability that the null is true, an effect-size estimate or a guarantee of practical importance. Report the observed difference and the design separately from the tail probability.
Update the test when an assignment error, missing outcome, wrong block identity or statistic definition is corrected. Preserve the old allocation table and explain the change. The audit should make clear whether the evidence changed because a legal assignment was recovered, a null assumption changed, or only the number of Monte Carlo draws increased.
Check what the small design can resolve
The six equally probable assignments also limit attainable p values. With this statistic, no lower-tail result can have p below 1/6. More decimal places, or a million simulated draws, cannot create a smaller exact tail in this experiment. A protocol seeking a conventional 0.05 threshold would need a suitable larger design or a different justified decision framework before data collection.
Build the enumeration from document identities, not from sorted outcomes alone. For each treated subset, take its complement as control, verify that both contain two documents, then calculate the two means using the same outcome file. Check that every document appears in three treated subsets and that complementing treatment reverses the statistic. These checks recover six rows, two zeros and a distribution symmetric around zero without assuming symmetry as a software setting.
If the allocation probabilities are missing, the table still shows possible statistics, but equal row weighting is an unverified design claim. Recover the allocation code or protocol rather than inferring probabilities from the one assignment that happened. A record of “two selected” does not establish that every pair was equally likely.
FAQ
Why must the observed assignment be counted?
It is one assignment allowed by the design and is at least as extreme as itself. Excluding it understates the exact tail probability.
Can arbitrary permutation make an observational comparison randomized?
No. A valid reference distribution needs a justified assignment or exchangeability model; shuffling alone supplies neither.