Measurement

In plain words

Suppose 1,000 people use a website and 600 of them put "housing" near the top of their list. That tells you about those 1,000 people. It does not tell you what everyone in a city thinks, because the people who chose to show up may be different from the people who did not.

Researchers have ways to adjust for this, but none of them is magic. Large numbers do not fix it. A big group of self-selected people can be very precisely wrong.

So this framework treats every number as a statement about "these participants, this question, this time." It shows who is counted, who is missing, how the number was calculated, and what changed when a number is revised. How questions are worded is covered in Question design. How individual answers are combined into a group priority is covered in Collective priorities.

Defined terms

Population. The full group a result is meant to describe, such as registered voters in a county.

Participants. The people who actually answered. In an open website, participants choose themselves.

Denominator. The group a percentage is calculated over. "60% chose housing" means different things if the denominator is everyone who visited, everyone who started, everyone who finished, or everyone who answered that one item.

Probability sample. A sample in which every member of a defined population has a known, nonzero chance of selection. Nonprobability (opt-in) sample. A sample in which people select themselves or are selected without known probabilities¹.

Nonresponse. People selected or invited who do not answer. Nonresponse bias arises when those who answer differ from those who do not on the thing being measured².

Weighting. Adjusting how much each answer counts so that the participant group better matches known characteristics of the population³.

Ordinal ranking. A ranking that tells order (first, second, third) but not distance. The gap between rank 1 and rank 2 is not assumed equal to the gap between rank 2 and rank 3.

Distribution. The full spread of answers, rather than a single average.

Missing data. Answers that are absent, skipped, or unconfirmed. The 2020 AAPOR task force preferred the word "missingness" when the reason for absence cannot be fully known⁴.

Revision. A participant changing an earlier answer, or an operator correcting a published number.

See the glossary for the framework's definitions of ranking and effectiveness.

Evidence

Opt-in samples can be precisely wrong

Pew Research Center tested how much weighting can fix opt-in online samples. Across 24 benchmarks, unweighted opt-in estimates were off by an average of 8.4 percentage points. No adjustment procedure tested brought average bias below 6 points; the best removed only about 30% of the original bias³. Pew found that "choosing the right variables for weighting is more important than choosing the right statistical method"³. Increasing the sample from 2,000 to 8,000 shrank the modeled margin of error from about plus or minus 1.8 points to 0.5 points, but average bias held at 6.3 points, so estimates became "more tightly clustered around the wrong (biased) value"³.

An earlier Pew study compared nine opt-in samples from eight vendors against Pew's own probability panel on the same questions. It found that "online nonprobability surveys are not monolithic": one vendor outperformed the others, including Pew's panel, on nearly all measures, while estimates for Hispanic and Black adults had average bias of 15.1 and 11.3 points. All the online samples contained more civically and politically engaged people than benchmarks indicated, with very large bias on volunteering and community problem solving⁵. The probability panel itself tied for the largest bias on civic engagement items⁵, a reminder that probability methods also have limits.

In 2024 Pew reported a further problem: "bogus respondents" who answer insincerely to collect rewards. In one opt-in poll, 20% of adults under 30 agreed that "the Holocaust is a myth," while Pew's probability panel found 3% in that age group. In another opt-in sample, 12% of respondents under 30 claimed to hold a license to operate a nuclear submarine⁶. Such answers inflate rare views and concentrate in some groups⁶.

Nonresponse does not always mean bias, but it raises risk

Pew reports that typical telephone survey response rates fell to 7% in 2017 and 6% in 2018, and argues that response rates are "an unreliable metric of accuracy": bias depends on whether the thing measured is related to the decision to respond⁷. Groves's review in Public Opinion Quarterly reaches a compatible conclusion: "Nonresponse can, but need not, induce nonresponse bias," and the value of adjustment depends on the quality of auxiliary information². Both sources note topics such as volunteering where nonresponse and engagement are linked⁷⁵. Participation in civic exercises is plausibly one of those topics. That is an interpretation, but it is the reason this framework treats civic engagement as the variable most likely to separate participants from nonparticipants.

No single theory covers opt-in inference

The AAPOR Task Force on Non-Probability Sampling concluded in 2013 that "there is no single framework that adequately encompasses all of non-probability sampling," that "making inferences for any probability or non-probability survey requires some reliance on modeling assumptions," that those assumptions should be stated, and that "transparency is essential," with a higher disclosure burden for nonprobability methods¹.

Missingness can defeat explanation

After the 2020 U.S. election, an AAPOR task force found polling error of "unusual magnitude," the highest in 40 years for the national popular vote. It ruled out several explanations but concluded that identifying the cause conclusively "appears to be impossible with the available data," because reliable information is lacking on those who were not included or chose not to take part⁴. The lesson for any participation system is direct: once the missing people are missing, their views cannot be reconstructed from the people who are present.

Disclosure is a professional standard

The AAPOR Code of Professional Ethics and Practice (revised April 2021) requires immediate disclosure of, among other items, the sponsor, question wording and response options, population under study, sample recruitment method, mode, dates, sample sizes, weighting method and sources, data processing and exclusions, and "a general statement acknowledging limitations"⁸. AAPOR's Standard Definitions (tenth edition, 2023) standardizes how case outcomes and response rates are coded⁹.

Competing interpretations

Participation results should be weighted to look like the public. Weighting makes numbers more comparable to population benchmarks, and Pew's work shows it removes some bias³.

Participation results should not be presented as public opinion at all. Weighting removed only a minority of bias in Pew's test³, and in a civic setting the most important difference between participants and others, engagement, may not be measurable. On this view, an open participation site should describe its participants honestly and stop there.

Large numbers confer legitimacy. A frequent public intuition is that 100,000 responses must mean something.

Large numbers confer only precision. Pew's finding that larger opt-in samples cluster more tightly around a biased value³ answers this directly.

Averages summarize rankings well. Mean rank is easy to read.

Distributions are more honest. An average rank can hide a split where half put a subject first and half put it last. Ordinal data does not support treating rank gaps as equal distances, which is why the framework shows distributions and ties. Aggregation rules themselves are compared in Collective priorities.

Application to this framework

These are design proposals, not results.

1. State the denominator in the sentence. Not "62% prioritize water," but "62% of 1,140 participants who confirmed a ranking between dates X and Y placed water in their top three." Unconfirmed defaults are excluded and counted separately (see Priorities).
2. Describe participants, not the public. Unweighted participant results are labeled as such. Any weighted figure is labeled as modeled, lists its weighting variables and sources, and is shown beside the unweighted figure, following AAPOR disclosure items⁸.
3. Show distributions and ties. Rankings are shown as distributions and as rank intervals where the method supports them. Near ties are shown as ties (see Question design on false precision).
4. Report missingness. Each release states how many started and did not finish, how many skipped each item, and how many answered "not sure."
5. Treat coordinated or insincere participation as a measurement risk. Pew's bogus respondent findings⁶ show that incentives and anonymity can attract insincere answers. The framework does not pay for participation, and integrity checks are described in Security and Identity and registration without claiming they eliminate the problem.
6. Version revisions. A participant who changes a ranking creates a new dated record; published results state which snapshot they use. Operator corrections are logged and visible, as described in Published releases.
7. No individual political profiles in measurement. Aggregates are computed under the privacy rules in Privacy; small groups are suppressed according to a documented minimum group size that is a provisional design choice, not yet set.
8. No inference to mandates. A participant distribution is an input to discussion. It is not a vote and does not bind any authority (see Representation).

Hypothetical example. A town's demonstration exercise collects 5,000 rankings, 70% from one neighborhood that held an organized sign-up event. The framework's release would report the geographic distribution of participants, show results by area where groups are large enough, and say plainly that the overall figure overrepresents one neighborhood. It would not silently reweight the result and present it as the town's view.

Unresolved questions

• Can a civic participation system ever obtain trustworthy population benchmarks for engagement, the variable most likely to drive bias?
• Should a framework run a small probability-based companion survey to calibrate open participation, and who would fund and govern it independently?
• What minimum group size protects privacy without hiding small communities' priorities? This is a privacy and governance decision still to be made.
• How should revised rankings be weighted in time series: does a person who updates often count more?
• How can a release show uncertainty for opt-in data when standard margins of error do not apply?