Opt-in panels: the mathematics of self-selection and the limits of adjustment

An opt-in panel is a pool of people who volunteered to take surveys, usually for small rewards, rather than being selected at random. Statistician Xiao-Li Meng showed that the error of any such estimate equals a data quality term times a data quantity term times a difficulty term, so a tiny correlation between who participates and what they think can make millions of answers no more accurate than a few hundred random ones. Adjustment helps only to the extent that the reasons people join are measured.

In plain words

When people pick themselves, the question is not “how many answered?” but “are the people who answered different, on this question, from the people who did not?”

If the difference is tiny, a huge sample can still be badly off, because size multiplies the effect of that tiny difference. Researchers have shown this with real surveys: in 2021, two very large surveys overstated how many American adults had been vaccinated, while a much smaller, carefully recruited panel got it about right.

Statisticians can adjust opt-in data to look more like the population on age, education and similar traits. That helps when those traits explain why people joined. It does not help when the reason people joined is the very thing being measured. This matters directly for a civic site, where the people most interested in politics are the most likely to show up.

Measurement already summarizes Pew Research Center’s tests of weighting and its findings on bogus respondents. This page explains the mathematics underneath those findings and the newer evidence, and it does not repeat them. Probability sampling is explained in Survey sampling.

Defined terms

• Opt-in (nonprobability) panel. A pool of volunteers recruited through ads, websites or partner lists, with unknown selection probabilities. See the glossary entry for opt-in sample.
• River sampling. Recruiting respondents on the fly from banner ads and pop-ups; AAPOR describes it as recruitment “using banner ads, pop-up ads and similar instant ‘capture’ promotions”.⁠⁶
• Participation rate. For nonprobability samples, the share of invited panel members who give a usable response. AAPOR says this “should not be referred to as a ‘response rate’”.⁠⁶
• Data defect correlation (ρ_R,X). The correlation, across the whole population, between whether a person is recorded in the data (R) and the value being measured (X).⁠¹
• Quasi-randomization. Estimating “pseudo-inclusion probabilities” from covariates and weighting by their inverse.⁠³
• Superpopulation modeling. Fitting a model for the outcome and using it “to predict values for the nonsample units”.⁠³
• Raking. Iteratively adjusting weights until weighted sample margins match known population margins, a method descended from Deming and Stephan’s 1940 work on adjusting a sampled frequency table “when the expected marginal totals are known”.⁠⁹
• Multilevel regression and poststratification (MRP). Modeling the outcome within many small demographic cells, then averaging cell estimates using population counts.⁠⁴
• Insincere (bogus) respondent. A respondent who answers without reading or honestly considering questions, often to collect rewards.⁠⁸

Evidence

Meng’s identity: quality, quantity, difficulty

In a 2018 paper in the Annals of Applied Statistics, Meng showed that “for any dataset of size n, probabilistic or not, the difference between the sample average and the population average is the product of three terms”: a data quality measure ρ_R,X; a data quantity measure sqrt((N-n)/n), where N is the population size; and a problem difficulty measure σ_X, the standard deviation of X.⁠¹ In plain text:

error = rho_RX * sqrt( (N-n)/n ) * sigma_X

The identity is exact, not an approximation. Its consequences, as Meng states them:⁠¹

• Random sampling controls ρ at the level of 1/sqrt(N), which is what makes small random samples work.
• Without that control, error relative to the usual 1/sqrt(n) benchmark “increases with sqrt(N),” which Meng calls the Law of Large Populations.
• The “bigness” of data for population inference “should be measured by the relative size f=n/N, not the absolute size n.”
• When combining sources, “relatively tiny but higher quality ones should be given far more weights than suggested by their sizes.”

Meng’s worked example uses the 2016 Cooperative Congressional Election Study. He estimates ρ ≈ -0.005 for self-reporting a vote for Donald Trump, and shows that this “seemingly minuscule data defect correlation implies” that a self-reported sample of about 2,300,000 people, 1% of eligible voters, “has the same mean squared error as the corresponding sample proportion from a genuine simple random sample of size n≈400”.⁠¹ He calls the result a Big Data Paradox: “the more the data, the surer we fool ourselves.”

The identity reframes the familiar finding, reported in Measurement, that larger opt-in samples cluster more tightly around a wrong value. Increasing n shrinks the random part of error but leaves ρ untouched, and for a fixed population the quantity term falls only slowly.

A natural experiment: three surveys and one benchmark

Bradley, Kuriwaki, Isakov, Sejdinovic, Meng and Flaxman, writing in Nature in 2021, compared three surveys of COVID-19 vaccine uptake among U.S. adults against the Centers for Disease Control and Prevention’s count of first doses.⁠² In May 2021, “Delphi-Facebook overestimated uptake by 17 percentage points” and the Census Household Pulse survey “by 14,” each with ranges reported under an assumed 5% benchmark imprecision.⁠² Delphi-Facebook collected “about 250,000 responses per week” and Household Pulse “about 75,000 every two weeks,” while “an Axios-Ipsos online panel with about 1,000 responses per week” was, in the authors’ words, reliable.⁠² The authors show “how a survey of 250,000 respondents can produce an estimate of the population mean that is no more accurate than an estimate from a simple random sample of size 10”.⁠²

Two further points from the article bear on any civic platform. The authors conclude that the growing error in the two large surveys was primarily driven by an increasing data defect correlation, and they state that “weighting on observed covariates alone cannot explain or correct the discrepancies we observe”.⁠² Note that Household Pulse is a Census Bureau survey drawn from an address frame, not a volunteer panel. Its very low response made it behave, on this outcome, like a self-selected sample. The lesson is about selection on the outcome, whatever the recruitment channel.

Two families of correction

Elliott and Valliant’s 2017 review in Statistical Science sets out the two main approaches when “design-based inference ... cannot be used for nonprobability samples”:⁠³

1. Quasi-randomization. Combine the opt-in sample with a reference probability sample, model the probability that a unit with covariates x appears in the opt-in sample, and weight each opt-in case by the inverse of its estimated propensity. This works if, conditional on x, joining the panel is unrelated to the outcome.
2. Superpopulation modeling. Fit a model of the outcome y given x on the opt-in cases, then predict y for everyone in the population using known distributions of x. This works if the model holds for people who did not join.

They discuss “the pros and cons of each approach”.⁠³ The shared assumption is the important part. Both are versions of “missing at random given x.” Neither can test that assumption with the opt-in data alone.

MRP: the Xbox poll

The best-known demonstration of model-based correction is Wang, Rothschild, Goel and Gelman’s study of daily voter intention polls run on the Xbox gaming platform during the 2012 U.S. presidential campaign.⁠⁴ The data comprised 750,148 interviews with 345,858 unique respondents. The sample was badly skewed: “18- to 29-year-olds comprise 65% of the Xbox dataset, compared to 19% in the exit poll; and men make up 93% of the Xbox sample but only 47% of the electorate”.⁠⁴

The authors partitioned the population “into 176,256 cells” defined by demographic and political variables, including party identification and 2008 vote, estimated vote intention in each cell with a multilevel model that lets sparse cells borrow strength from similar ones, and combined cells with the poststratification estimator:⁠⁴

y_PS = sum over cells j of ( N_j * y_hat_j ) / sum over cells j of N_j

After adjustment, they report estimates “in line with the forecasts from leading poll analysts, which were based on aggregating hundreds of traditional polls”.⁠⁴

Two details limit how far the result travels. First, the authors stress that “including the political variables is important because they are strong predictors of vote intentions”.⁠⁴ Pew’s 2018 weighting study agrees: for political engagement benchmarks, unweighted bias averaged 22.3 points, demographic weighting reduced it by 2.9 points, and adding political adjustment variables reduced it by 11.7 points, to 10.6.⁠⁷ Pew also found that more complex methods “never reduced the average estimated bias by more than 0.3 points beyond what was achieved with raking”.⁠⁷ Second, vote intention in a two-candidate race is strongly predicted by party identification. Priorities across ten public subjects may not be predicted as well by anything a platform can measure.

Panels are not one thing

Pew’s 2023 comparison of three probability-based panels and three opt-in samples across 28 benchmarks found average absolute errors of 2.6 points and 5.8 points respectively.⁠⁵ The gap widened for subgroups: on opt-in samples, 18- to 29-year-olds averaged 11.2 points of error and Hispanic adults 10.8 points, against 3.6 points for both groups on probability panels.⁠⁵ Pew attributed much of the opt-in error to “bogus respondents”: in opt-in samples, 8% of all adults, 15% of 18- to 29-year-olds and 19% of Hispanic adults answered “Yes” on at least 10 of 16 yes or no questions, compared with 1% to 2% on probability panels.⁠⁵

Insincere respondents do not answer at random

Kennedy and colleagues, writing in Public Opinion Quarterly in 2021, studied six online sample sources, including three opt-in panels, two address-recruited panels and a crowdsourced sample, and analyzed 375,834 open-ended answers to detect insincere respondents.⁠⁸ They found that incidence “varied significantly by the type of online sample,” and that insincere respondents “tended to select positive answer choices, introducing a small, systematic bias”.⁠⁸ Random noise would wash out in a large sample. A consistent lean toward “yes” does not; by Meng’s identity it is exactly the kind of correlation that scale amplifies.

Response rates do not apply

AAPOR’s Standard Definitions states that for opt-in panels “the number of people who were exposed to the invitation, and the number of invitations to which they were exposed, are not known,” that a participation rate “should not be referred to as a ‘response rate’ because, unlike for probability-based samples, a high response rate does not necessarily mean the risk of bias is reduced,” and it cautions “strongly” against presenting such metrics as indicators of quality.⁠⁶

Competing interpretations

“Adjusted opt-in data is good enough, and sometimes better.” The Xbox result,⁠⁴ and the opt-in samples’ closer turnout estimates in 2023⁠⁵ support the view that well-adjusted volunteer data can match or beat low-response probability surveys on some outcomes, faster and at lower cost.

“Adjustment is unfalsifiable without a benchmark.” Bradley and colleagues’ finding that covariate weighting could not correct the vaccine surveys⁠² supports the opposite view: the cases where adjustment works are the ones where a benchmark exists to prove it, and those are exactly the cases where the survey was least needed.

“Size is not worthless.” Large opt-in data can reveal within-sample patterns, such as how answers relate to each other or change over time among the same people, that do not require population inference. Critics accept this but warn that change over time can also be distorted if who participates changes over time.

“Fraud is a vendor problem.” Some treat insincere respondents as a quality control issue to be filtered. Others argue, citing their systematic lean⁠⁸ that filtering is always partial and that incentive-driven recruitment creates the problem it then tries to filter.

Application to this framework

These are design proposals for My Vote Forward, not results.

1. Treat open participation as an opt-in sample with ρ unknown. Every release says that the data defect correlation for civic priorities is unknown and cannot be estimated from participant data alone.
2. No rewards for participation. Kennedy and colleagues’ and Pew’s findings on insincere respondents,⁠^8,5 concern incentive-driven panels. Not paying for participation removes one motive, though not every motive. Coordinated campaigns are treated separately in Coordinated campaigns detection.
3. Model-based estimates only beside a benchmark. If an institution asks for a population estimate, any MRP or propensity-weighted figure appears only alongside an independent probability benchmark for the same question, with the model, cell definitions, covariates and population totals published. Without a benchmark, the release shows participant figures only.
4. Measure engagement, and say why. Because political engagement variables did the most work in Pew’s adjustment study⁠⁷ a companion probability survey would need to measure engagement in a way that could be matched to participants. That raises privacy questions, covered in Privacy.
5. Weight small, high-quality data heavily. Following Meng,⁠¹ if a small probability survey and a large open exercise disagree, the release treats the small survey as the better estimate of the population and the large exercise as a description of participants.

Hypothetical example. A city’s open exercise draws 20,000 participants from a population of 400,000 adults (f = 0.05). Suppose, purely for illustration, that the correlation between participating and ranking public transit first is 0.02, and the standard deviation of that yes or no outcome is 0.45. By Meng’s identity the error in the share ranking transit first is about 0.02 times sqrt(380,000/20,000) times 0.45, roughly 0.039, or 3.9 percentage points, before any random error. A simple random sample of about 130 people would have a standard error of the same size (0.45 divided by the square root of 130 is about 0.039). The participant count sounds a hundred times larger. Its population accuracy, under this assumed correlation, is not.

Unresolved questions

• What benchmark exists for civic priorities? Unlike vaccine doses or vote counts, there is no administrative count of what a public thinks matters most, which means opt-in priority data may never be validated directly.
• Could a platform estimate ρ for a few questions by embedding them in an independent probability survey, and would that estimate transfer to other questions?
• If participation changes over time in who shows up, how should trends be reported so that a change in the participant mix is not read as a change in public priorities?
• Is there an ethical way to measure political engagement for adjustment without building political profiles of participants?
• Should releases show a “Meng equivalent” sample size under stated assumptions, or would that invite false precision of its own?