Ranked preferences: statistical models for lists of what matters most

Ranked preference data are answers in which each person orders a set of items, such as public subjects, from most to least important. Statisticians model such data with tools built for orderings: distances between rankings such as Kendall’s, consensus rules such as Kemeny’s, and probability models such as Mallows and Plackett-Luce. Each tool answers a slightly different question, and none removes the need to show how uncertain a published order is.

In plain words

If you ask people to rank ten subjects, you get lists, not numbers. Lists are tricky. Saying that “health” averaged 3.2 and “housing” averaged 3.4 treats the gap between first and second place as if it were the same as the gap between ninth and tenth. It may not be.

Statisticians have better tools. Some measure how far apart two lists are by counting how many pairs of items they disagree about. Some find the single list that disagrees least with everyone. Some imagine that each person builds a list by picking a favorite, then a favorite among what is left, and estimate how strong each item’s pull is.

All of these tools give an order. None of them tells you, by itself, how sure you can be about that order. A subject in fourth place might easily be second or seventh with slightly different participants. Honest results show that range.

The problems of combining votes, such as Condorcet cycles and Arrow’s theorem, are covered in Collective priorities. This page covers the statistics of rankings themselves.

Defined terms

• Full ranking. An ordering of all k items. With k = 10 there are 10! = 3,628,800 possible rankings.
• Top-t (partial) ranking. An ordering of only the first t items, with the rest unranked.
• Kendall tau distance. The number of item pairs on which two rankings disagree. It ranges from 0 to k(k-1)/2.
• Kemeny consensus. The ranking that minimizes the total Kendall distance to all submitted rankings.
• Mallows model. A probability model in which rankings closer to a central ranking are more likely.
• Plackett-Luce model. A probability model in which a ranking is built by choosing items one at a time, each with probability proportional to a positive worth parameter.
• Bradley-Terry model. A model for paired comparisons in which item i beats item j with probability α_i/(α_i+α_j).
• Best-worst scaling (BWS). Asking people to pick the most and least important items from a small subset.
• Ipsative data. Scores that must sum to a fixed total for each person, as ranks do, which forces negative dependence among items.
• Rank interval. A range of ranks an item plausibly occupies, given sampling or measurement uncertainty.

Evidence

Measuring distance between rankings

Maurice Kendall introduced “a new measure of rank correlation” in Biometrika in 1938.⁠¹ The associated distance counts discordant pairs. For two rankings π and σ of the same k items:

d_K(pi, sigma) = number of pairs (i, j) ranked in opposite order by pi and sigma

and the correlation is tau = 1-4 d_K / ( k(k-1) ), which runs from -1 (exact reverse) to +1 (identical). For k = 10 there are 45 pairs. Two participants who agree on 40 pairs and disagree on 5 have tau = 1-20/90, about 0.78.

Kendall distance has a property that suits public priorities: swapping two adjacent items costs 1 no matter where in the list they sit. Some analysts prefer weighted distances that make disagreement at the top count more, since the top of a priority list usually matters most. That choice is a value judgment and should be stated.

Kemeny’s rule as a statistical estimate

H. Peyton Young’s 1988 article in the American Political Science Review reinterpreted Condorcet’s work as statistics. Young writes that Condorcet argued majority choice was “statistically most likely to be the best choice” if voters make independent errors; that this claim “is not true” in every situation; but that Condorcet “did propose a novel and statistically correct rule for finding the most likely ranking of the alternatives,” the procedure “sometimes known as ‘Kemeny’s rule’”.⁠² Young adds that it is “the unique social welfare function that satisfies a variant of independence of irrelevant alternatives together with several other standard properties”.⁠²

In plain text, the Kemeny consensus is:

pi_star = argmin over all rankings pi of ( sum over participants n of d_K(pi, pi_n) )

Two consequences matter for practice. First, the Kemeny consensus is a maximum likelihood estimate under a specific error model, so it has a clear meaning: the order most consistent with all pairwise preferences. Second, a brute-force search must compare every possible ordering, and the number of orderings grows as k factorial (3,628,800 for ten items), so exact computation needs care as lists get longer. When majority preferences cycle, as described in Collective priorities, the Kemeny consensus still exists but rests on fragile margins, which is a reason to publish it with its pairwise majority table.

Mallows: a probability model around a consensus

Colin Mallows published “Non-null ranking models” in Biometrika in 1957.⁠³ The model that bears his name states, in its most common form:

P(pi) proportional to exp( -theta * d_K(pi, pi_0) )

where π_0 is a central ranking and θ ≥ 0 measures how tightly people cluster around it. When θ = 0 every ranking is equally likely; as θ grows, rankings far from π_0 become rare. The maximum likelihood estimate of π_0 under this model is the Kemeny consensus, which ties the probability model to the consensus rule above.

A single Mallows model assumes one center. Public priorities often have several: groups that agree internally and disagree with each other. Mixtures of Mallows models, with one center per latent group, can describe that structure. That is a modeling choice with political weight, because naming the groups is an interpretation the data do not make on their own.

Plackett-Luce: rankings as a sequence of choices

Robin Plackett’s 1975 paper “The Analysis of Permutations” defined “a probability distribution ... over the r! permutations of r objects” and applied it to “data on voting at elections”.⁠⁴ The model, now called Plackett-Luce, gives each item a positive worth α and treats a ranking as a sequence of choices from what remains:

P(pi) = product over positions s = 1..t of ( alpha_pi(s) / sum over items not yet chosen of alpha )

Three features make Plackett-Luce useful for civic priorities:

1. It handles top-t rankings naturally. The product simply stops at position t. A participant who ranks only their top three contributes exactly three factors.
2. Its parameters have a direct reading. The worth ratio α_i/α_j is the odds that i is chosen over j when both are available.
3. It inherits a strong assumption. At each stage, the relative odds of two remaining items do not depend on which other items remain. If adding a new subject such as “public safety technology” draws mostly from “safety and justice,” that assumption fails, and estimated worths can shift in ways the model cannot represent.

Bradley-Terry: when people compare pairs

Bradley and Terry’s 1952 Biometrika paper analyzed “the method of paired comparisons”.⁠⁵ If participants see two subjects at a time and choose the more important, the model is:

P(i preferred to j) = alpha_i / ( alpha_i + alpha_j )

Pairwise designs are easier for participants than full rankings and can be spread across many short sessions. David Hunter’s 2004 paper in the Annals of Statistics showed how “minorization-maximization” algorithms produce maximum likelihood estimates for “a wide class of generalizations of the Bradley-Terry model,” with “simple conditions ... that guarantee that each algorithm described will produce a sequence that converges to the unique maximum likelihood estimator”.⁠⁶ One practical condition is connectivity: if some subjects are never compared, directly or through a chain, their relative worth cannot be estimated.

Best-worst scaling

Louviere, Flynn and Marley describe best-worst scaling as “an extension of the method of paired comparison to multiple choices that asks participants to choose both the most and the least attractive options or features from a set of choices,” and argue it “avoids many rating scale problems”.⁠⁷ For ten subjects, a design might show subsets of four, repeated so that every subject appears equally often. A simple score for each subject is the count of “most important” choices minus the count of “least important” choices, divided by appearances. BWS asks for fewer judgments per screen than a full ranking, which may suit mobile participants. Whether it changes the resulting order compared with full ranking, for civic subjects, is an empirical question this page cannot answer.

Rankings versus ratings

Rating each subject on a scale and ranking the subjects are different tasks. Alwin and Krosnick, using the 1980 General Social Survey, found the two methods “similar with respect to ordering the aggregate preferences of the sample, but dissimilar with regard to the latent variable structure underlying the measures”.⁠⁸ Moors, Vriens, Gelissen and Vermunt, using latent class models in a within-subjects design, concluded that “ranking and ratings do produce results that are more similar than often thought”.⁠⁹ Hino and Imai report it is “widely accepted that factor analysis based on ranking scales suffers from a negativity bias, while that based on rating scales in turn suffers from a positivity bias”.⁠¹⁰ The negativity comes from ipsativity: because one person’s ranks always sum to the same total, raising one subject must lower another, which creates negative correlations among subjects that say nothing about real opposition between them.

For a priority platform this means that aggregate orders from ratings and rankings may agree, while analyses of how priorities group together (which subjects “go with” which) can differ by method.

How certain is a rank?

Goldstein and Spiegelhalter’s 1996 paper in the Journal of the Royal Statistical Society, Series A, titled “League Tables and Their Limitations: Statistical Issues in Comparisons of Institutional Performance,” is a standard reference on the danger of publishing ordered tables without uncertainty.⁠¹¹ Hall and Miller’s 2009 study in the Annals of Statistics sharpened a technical point: “the conventional bootstrap can produce inconsistency” for empirical ranks, and “the standard bootstrap gets the order of magnitude of the interval right, but not the constant multiplier of interval length”; an “m-out-of-n bootstrap can improve performance” but “requires empirical choice of m”.⁠¹²

The practical upshot is that rank intervals are informative but imperfect, and items whose underlying worths are nearly equal will have wide, overlapping intervals. The near ties are real findings and should be reported as ties.

Hypothetical example. In 5,000 top-three rankings of ten subjects, “housing” is chosen first by 19% and “health” by 18%. A Plackett-Luce fit gives housing a slightly higher worth. Resampling participants shows housing ranked first in 58% of resamples and health in 42%. A defensible release says the two are statistically tied for first and shows both rank intervals, rather than announcing housing as “the public’s top priority.”

Competing interpretations

“Use the consensus with the best axioms.” Young’s argument for the Kemeny rule,⁠² supports publishing the ranking most consistent with all pairwise preferences. Critics note that the rule is hard to explain and can change when a single subject is added, depending on which axiom one relaxes.

“Use a probability model, because it gives uncertainty.” Plackett-Luce and Mallows give likelihoods, standard errors and goodness of fit. Critics answer that every model makes structural assumptions, such as Plackett-Luce’s independence from irrelevant alternatives, that may not hold for public subjects that overlap.

“Do not model; show the data.” Counts of first choices and top-three shares are transparent and need no model. Critics reply that simple counts discard information from lower positions and can produce different orders than full-ranking methods.

“Ratings are more informative than rankings.” Ratings allow ties and intensity. Supporters of ranking reply that ratings let people say everything is very important, which avoids the tradeoff the exercise is meant to reveal. See Tradeoffs.

Application to this framework

These are design proposals for My Vote Forward, not results.

1. Publish three views of the same data. For any priority release, show first-choice and top-three shares (no model), a Kemeny or Plackett-Luce order (stated model), and the full pairwise majority table. Agreement among the three is evidence of a stable order; disagreement is itself a finding.
2. Show rank intervals and declare ties. Use resampling with the method disclosed, mindful of Hall and Miller’s caution⁠¹² and report items whose intervals overlap substantially as tied.
3. Fix the subject list per release and version it. Because Plackett-Luce worths and Kemeny orders can change when items are added, a change in the subject list starts a new series. See Who chooses the ten.
4. Avoid factor analysis of ranks without correction. Following the ipsativity problem,⁠¹⁰ the framework would not publish claims that two subjects are “opposed” based on negative correlations among ranks.
5. Test alternative formats before adopting them. Pairwise comparison and best-worst scaling are candidates for mobile use; the framework would run a comparison against full ranking in a demonstration before any switch, as described in Evaluation.

Unresolved questions

• Should disagreement near the top of a list count more than disagreement near the bottom, and who should decide that weighting?
• Is Plackett-Luce’s independence assumption reasonable for public subjects that overlap, such as housing and economy, and how could it be tested in a demonstration?
• How should mixtures of rankings (distinct groups with distinct centers) be reported without labeling groups in ways that invite partisan readings?
• What resampling method gives reliable rank intervals for ten items and a few thousand participants, given known bootstrap problems⁠¹²?
• Would best-worst scaling or pairwise comparison change participants’ orders compared with full ranking for civic subjects?