What if most people never participate?

Then the results describe the people who took part, not the public. That is a central limit of any opt-in system, including My Vote Forward, because people who choose to take part often differ from people who do not. The framework’s answer is disclosure rather than correction by assertion: every published aggregate must state who took part, the denominator, how missing answers were handled, and that participants are not the same as constituents. Low participation also weakens any claim that a result should guide policy.

The objection in its strongest form

The objection is that a framework built on voluntary participation will produce results that are systematically unrepresentative, and that disclaimers will not stop people from treating them as the public’s view. People with strong opinions, free time, internet access, and confidence in civic institutions will take part. People who are busy, disengaged, distrustful, or without reliable access will not. The resulting aggregates will overstate some concerns and erase others, and the people erased will be those already least heard.

The strongest version adds that the problem cannot be fixed afterward. Statistical weighting can adjust for known differences such as age and region, but not for the reason people chose to take part, which is often related to the very opinions being measured. Worse, a framework that publishes regularly will become a target: activists will know that a few hundred mobilized participants can move a result that thousands of uninvolved people never see. In this view, an opt-in civic tool does not just fail to represent the public; it gives a misleading picture of it a legitimate-looking format.

Why the objection matters

The objection matters because My Vote Forward publishes collective priorities. If those priorities are quoted without their denominators, they will be mistaken for the public’s priorities. It also matters for the framework’s purpose. A tool meant to make participation between elections more informative could make it less informative if the people who use it are a narrow slice and their results are treated as general.

Defined terms

• Opt-in sample: a set of participants who chose to take part, as opposed to people selected at random.
• Probability sample: a sample in which every member of a population has a known chance of selection.
• Nonresponse bias: distortion that arises when people who do not respond differ from those who do.
• Weighting: adjusting results so the sample matches known population characteristics.
• Denominator: the number of people a percentage is based on. See measurement.
• Participant versus constituent: a participant took part; a constituent is anyone in the population. See representation.

Evidence

The classic failure. The 1936 Literary Digest poll is the standard cautionary example in survey research. Peverill Squire’s 1988 study, using a 1937 Gallup survey that asked people whether they had taken part in the Digest poll, concluded that the magazine’s “wildly incorrect estimate” resulted from two problems working together: the sample was skewed, and the people who responded were also skewed.⁠¹ Squire also argued that if everyone polled had responded, the poll would at least have picked the right winner.⁠¹ The lesson is that nonresponse alone can distort a large sample.

Professional guidance on opt-in panels. A task force of the American Association for Public Opinion Research concluded in 2010 that “Researchers should avoid nonprobability online panels when one of the research objectives is to accurately estimate population values,” adding that “There currently is no generally accepted theoretical basis from which to claim” that such results project to the population.⁠² The report also records that one task force member, Doug Rivers of Stanford University, “asked that his name be removed from the list of authors due to his disagreement with the conclusions of the report”.⁠² The conclusion was contested inside the profession at the time.

Current evidence on data quality. Pew Research Center’s 2026 methods report, titled “No Easy Fix for Bogus Respondents in Online Opt-In Polls,” screened an online opt-in survey of 11,114 U.S. adults, conducted November 14 to 19, 2024, using trap questions, automated prescreening, and voter file matching; 9,151 respondents passed the trap questions.⁠³ Failing a check does not prove a response was bogus, and the report should not be read as giving a single bogus-response rate. It does show that opt-in samples require extensive screening and that screening choices change who remains.

Disclosure standards. AAPOR’s disclosure standards require researchers to “Explicitly state whether the sample comes from a frame selected using a probability-based methodology,” to describe how weights were calculated, and specify that reports of nonprobability surveys will only provide measures of precision if they are defined.⁠⁴

How comparable systems have handled it

Comparable systems take three approaches. Survey research uses probability sampling where it needs population estimates, and discloses methods where it does not.⁠^2,4 Deliberative designs replace self-selection with random selection: Stanford’s Deliberative Polling begins with “A random, representative sample”,⁠⁵ and Ireland’s 2016 to 2018 Citizens’ Assembly members were “selected at random to represent the views of the people of Ireland”.⁠⁶ Participatory budgeting accepts self-selection and treats the result as a decision about a bounded budget made by those who took part; the Participatory Budgeting Project defines it as “a democratic process in which community members decide how to spend part of a public budget”.⁠⁷ That last approach does not claim representativeness; it claims legitimacy from an open process with real but limited stakes.

Competing interpretations

One interpretation holds that opt-in results are useful as signals of intensity. People who take part care enough to do so, and that intensity is politically real. On this view, the right practice is to publish opt-in results clearly labeled, never as public opinion, and let readers weigh them.

A second interpretation, in line with the AAPOR task force’s conclusion,⁠² holds that opt-in results should not be used to describe a population at all. On this view, if the framework wants to say anything about the public, it needs a probability sample or random selection.

A third interpretation holds that careful modeling can sometimes make nonprobability samples useful for population estimates. The AAPOR report itself records that one task force member disagreed with its conclusions,⁠² though the passage does not state his reasons. This page does not evaluate the modeling research; it notes only that the question was contested when the guidance was written.

What the framework answers

My Vote Forward’s rules are design proposals, not results:

• State the denominator in the sentence: “62% of 1,140 participants who confirmed a ranking between dates X and Y,” not “62% of people” (see measurement).
• Label unweighted results as describing participants. Any weighted figure is labeled as modeled, lists its weighting variables and sources, and is shown beside the unweighted figure.
• Report missingness: how many started and did not finish, how many skipped each item, how many answered “not sure.”
• Show the geographic and timing distribution of participation where groups are large enough, so concentration is visible.
• Never present a participant result as a mandate (see representation).
• Do not pay for participation, which reduces one known source of insincere answers.

Where the objection still wins

The objection wins on representativeness. The framework has no sampling design, no random selection, and no validated weighting method. Its results, if any real ones were ever collected, would describe participants only. Disclosure helps careful readers but does not stop careless or motivated ones from quoting a figure as the public’s view. And the framework has no evidence about who would actually participate, because it has collected no real participant data.

Unresolved questions

• Should a future deployment pair open participation with a random-sample panel to show how far they diverge?
• What minimum participation, if any, should be reached before a result is published at all?
• How could the framework reach people who are least likely to opt in without pressuring them?
• Should weighted estimates ever be published, given the professional disagreement about them?

Related

• Measurement
• Representation
• Collective priorities
• Is this just a poll?
• Can wealthy groups dominate?
• Hard questions