Opt-in sample: people who chose to take part

An opt-in sample is a group of respondents who chose to take part, for example by joining an online panel or answering an open web survey, rather than being selected at random from a defined population. Opt-in samples are cheap and fast, but research shows they can produce large errors when treated as representative. Every My Vote Forward participant is self-selected, so the framework reports results as describing participants, not the public.

Definition

The American Association for Public Opinion Research (AAPOR) Task Force on Nonprobability Sampling (June 2013) uses “the term opt-in panel to mean an online panel not recruited via probability sampling,” and stresses that “opt-in panels are not based on a single sampling method, but rely on myriad and varied sampling methods”.⁠¹ An opt-in sample is one kind of nonprobability sample. The task force notes that, “unlike probability sampling, there is no single framework that adequately encompasses all of non-probability sampling”.⁠¹

The contrast is with a probability sample, in which every member of a defined population has a known, nonzero chance of selection. That known chance is what lets researchers compute a margin of error.

Origin and history

Opt-in sampling spread with online research. The AAPOR task force writes that “non-probability sampling has become especially prevalent as more and more surveys have moved online”.⁠¹ It quotes an earlier AAPOR task force on online panels: researchers “should avoid nonprobability opt-in panels when a key research objective is to accurately estimate population values,” and “claims of ‘representativeness’ should be avoided when using these sample sources”.⁠¹ The 2013 report also concluded that the standard margin of error calculation is unsuitable for nonprobability samples, while warning that “treating estimates as though they had no error at all is not a reasonable option”.⁠¹

How opt-in samples are evaluated

Researchers evaluate opt-in samples by comparing their estimates with trusted benchmarks. Pew Research Center’s tests show both the problem and partial fixes. In a 2018 study of weighting online opt-in samples, Pew found that across 24 benchmarks, no adjustment procedure reduced average bias “below 6 percentage points,” down from 8.4 points unweighted, and that “even the most effective adjustment procedures were unable to remove most of the bias”.⁠²

A 2024 Pew analysis identified bogus respondents as a specific risk. In an opt-in survey, “12% of adults under 30” claimed to be licensed to operate a class SSGN submarine, while “the share of Americans with this type of submarine license rounds to 0%”.⁠³ Among adults under 30, 20% in the opt-in poll agreed that “the Holocaust is a myth,” compared with 3% in a probability-based panel.⁠³ Pew found such overreporting “especially concentrated in estimates for adults under 30, as well as Hispanic adults”.⁠³

These findings do not mean opt-in data is useless. They mean its errors are not described by a sampling formula, and they can be largest for exactly the groups a civic system most wants to hear from.

Common confusions and neighboring terms

• Opt-in vs. probability sample. Only a probability sample supports a conventional margin of error.
• Opt-in vs. participation. In a civic system, participants are by nature opt-in. Their views count as participation; they are not a poll of the public.
• Opt-in vs. nonresponse bias. Nonresponse affects probability samples when selected people decline. Opt-in samples have a different problem: selection is by the respondent from the start.
• Opt-in vs. weighting. Weighting can reduce some bias in opt-in samples but, per Pew, not most of it.⁠²

How My Vote Forward uses the term

My Vote Forward treats all participation as opt-in and labels results accordingly. Its Measurement chapter states that participants are not the whole public, and its releases must carry a denominator so readers can see who answered. The framework does not publish margins of error for participant results as if they were a probability survey. The demo uses simulated participants.

Unresolved questions

• Should the framework commission occasional probability surveys as a benchmark for participant results?
• How can bogus or careless responses be detected without collecting identifying information?
• When participant views differ sharply from probability survey results, which should institutions give weight to, and why?