Ordinal scores: preserve order without inventing equal distances
An ordered rating says which category comes before another. It does not, by itself, say how far apart the categories are. Before comparing average research scores, preserve the category definitions and ask what evidence makes their numerical spacing meaningful. Otherwise a change in coding can change the apparent winner while every underlying judgment stays the same.
The practical output is a versioned codebook, category counts, cumulative proportions and a record of any additional measurement assumptions. This is useful when comparing AI-generated evidence grades, analyst review labels or disclosure-quality rubrics. It evaluates the scale used to summarize judgments; it does not establish that the judgments themselves are correct.
Locate the measurement claim
Cochrane Handbook section 6.6 discusses ordered outcomes, measurement scales and alternative ways to analyze them. Its methodological discussion comes from evidence synthesis, not a validation of investment ratings. Here the research-workflow examples and recoding exercise are hypothetical.
Suppose a team assigns three evidence grades: incomplete, partly supported and supported. The ordering might reflect increasing fulfillment of a written requirement. Record that requirement first. If the categories instead mean different document types with no natural ranking, assigning them numbers does not create an ordinal scale.
Also record direction. A higher number could mean stronger evidence, greater concern or more work remaining. Mixing those meanings can reverse a comparison before any statistical issue arises. Keep missing, not applicable and not reviewed as separate states unless the codebook explicitly defines otherwise; none automatically belongs at the bottom of the evidence scale.
Recode the same observations
Consider two fictional review sets, each containing two excerpts. Set A receives incomplete and supported. Set B receives partly supported twice. No excerpt changes category in the following calculations.
| Category | Standard code | Alternative code | Another increasing code |
|---|---|---|---|
| Incomplete | 1 | 1 | −100 |
| Partly supported | 2 | 2 | 2 |
| Supported | 3 | 100 | 3 |
Under the standard coding, both means equal two: A is (1 + 3) / 2 and B is (2 + 2) / 2. Under the second coding, A becomes 50.5 while B stays two. Under the third, A becomes −48.5 while B again stays two. Every coding preserves the category order, yet the mean comparison moves from a tie to either direction.
The exercise does not show that an analyst made a multiplication error. It shows that a mean uses the numerical distances as well as the order. Choosing codes 1, 2 and 3 supplies a distance convention; the labels alone do not establish that the two steps represent equal increments in evidence strength.
A dashboard may still display a mean under an explicitly chosen scoring convention. Its claim should then be about that convention, with the coding and sensitivity visible. It should not describe a change from two to three as a measured 50% increase in evidential quality. The ratio depends on an arbitrary origin as well as spacing.
Preserve the distribution instead of hiding the tradeoff
The same observations can be reported without invented distances. Count each category and calculate proportions at each meaningful category boundary.
| Summary | Set A | Set B |
|---|---|---|
| Incomplete | 1 of 2 | 0 of 2 |
| Partly supported | 0 of 2 | 2 of 2 |
| Supported | 1 of 2 | 0 of 2 |
| At or below incomplete | 50% | 0% |
| At or below partly supported | 50% | 100% |
If higher is preferable, A has more top-grade excerpts and more bottom-grade excerpts. The cumulative comparisons cross. There is no single distributional ordering that makes A preferable at every threshold; the decision depends on which failure or success boundary matters. The data are also far too small to establish a reliable population difference.
Suppose the operational question is whether an excerpt can be used without additional evidence collection. If the codebook establishes supported as the required category, report one of two for A and zero of two for B. If the question instead concerns avoiding incomplete work, B does better in this sample. These are different outcomes, not competing arithmetic answers to the same question.
Choose a consequential threshold before inspecting which one favors the desired result. Retain all category counts so readers can see the cost of that choice. A threshold selected after examining outcomes should be labeled exploratory and evaluated on fresh material before making a wider performance claim.
Handle medians without interpolating fictional categories
A median category can be a useful location summary, but specify the convention. With an even number of observations, the two central categories may differ. In Set A they are incomplete and supported. Averaging their numerical codes produces a value whose interpretation changes under recoding; it does not identify an observed middle category.
Report the central-category interval, or state an explicit lower- or upper-median convention. Do not claim that every software implementation of a numerical median is automatically invariant to arbitrary increasing recoding. The ordered labels and a documented convention are what preserve the interpretation.
A rank-based comparison also answers its own question. It does not recover the missing measurement of distances. Ties, sample dependence and the observational unit still matter. Keep a record of excerpts, reports or issuers so that ten labels from one report do not quietly become ten independent issuers.
Use a different boundary: measured costs and validated scales
Now suppose the categories are linked to independently estimated follow-up costs of zero, five and one hundred monetary units per case. Averaging those costs could answer an expected-work-cost question if the estimates and population are justified. That calculation is no longer merely an average ordinal grade: it introduces a separate quantitative cost model.
Preserve the evidence for each cost, its date and uncertainty. A team cannot turn arbitrary codes into measured costs just by adding a currency symbol. If the expensive category becomes cheaper after a process change, update the cost model while preserving the original category decisions.
Using those hypothetical costs, A averages fifty monetary units and B averages five. If independently documented automation reduces the top category’s handling cost to eight while the other costs stay fixed, the averages become four and five. The cost comparison reverses because the quantitative process changed, not because the ordinal labels changed. Save the old and new cost assumptions rather than rewriting historical grades to manufacture the new result.
The same Cochrane section also discusses longer measurement scales and continuous-outcome summaries. It would therefore be too strong to declare all means of ordered scores invalid. A well-supported multi-item instrument may justify additional treatment; the justification concerns that instrument, its intended use and its measurement properties. Merely summing several unvalidated AI ratings does not supply that evidence.
Build a scale-audit worksheet
| Field | What to retain | Check before comparison |
|---|---|---|
| Ordered labels | Full definitions and examples | Is the order defensible? |
| Coding | Every numeric mapping and direction | Are distances assumed or measured? |
| Unit | Excerpt, report, issuer and date | Are observations repeated or clustered? |
| Distribution | Counts, missing and not applicable | Does a mean hide crossing categories? |
| Decision boundary | Threshold and reason | Was it chosen before outcomes? |
| Extra measurement | Cost or scale-validation evidence | Does it support the intended quantity? |
Retrieve the original rating rows rather than reverse-engineering counts from a rounded mean. If only the mean is disclosed, label the distribution unavailable. Many different category mixes can produce the same numerical average. Do not fabricate a typical mix to complete a comparison chart.
Keep the coverage denominator as well. Forty reviewed items and ten unreviewed items do not establish the grades of all fifty. Report the observed distribution among forty alongside the ten unresolved cases; coding the unreviewed cases as incomplete would create an additional judgment without reading their evidence.
Freeze the codebook version when comparing periods. If partly supported changes from requiring one corroborating source to requiring two, the category meaning has changed. Re-rate a common reference set or report the break in the series; simply mapping both versions to the number two does not make them comparable.
Reopen the assessment when definitions change, missingness grows, a new reviewer population is introduced or external evidence supports a quantitative scale. Save the earlier counts and assumptions. The defensible result states which category differences are observed, which decisions they support and which apparent score differences depend on an additional numerical convention.
FAQ
Does an ordinal scale forbid displaying a mean?
No. A mean can be reported under a clear convention, but stronger distance or quantitative claims need their own justification.
Does agreement between reviewers validate equal spacing?
No. Reviewers can apply the same categories consistently without establishing the numerical distance between them.