Survey sampling: how a few thousand answers can describe millions, and when they cannot

Survey sampling is the set of methods that let researchers describe a large population by questioning a small part of it. The justification comes from how people were selected: when every member of a defined population has a known, nonzero chance of selection, the sample’s answers can be weighted and an uncertainty range calculated. When selection is unknown, as in most open websites, those calculations do not apply.

In plain words

Imagine a pot of soup. You do not need to drink the whole pot to know whether it is salty. One spoonful is enough, but only if the soup is stirred. Random selection is the stirring. It is what makes a small sample look like the whole.

If you skim only from the top, a bigger spoon will not help. You will just get a more confident taste of the top layer. That is the problem with samples where people choose themselves.

This page explains how professional surveys do the stirring, how they calculate how wrong they might be, and why the people who never answer are the hardest part. The companion page on opt-in panels covers what happens when there is no stirring at all. The broader limits of any participation number are in Measurement, which this page does not repeat.

Defined terms

• Target population. The group a result is meant to describe, such as adults living in a county.
• Sampling frame. The list or mechanism from which a sample is actually drawn, such as a list of residential addresses. People missing from the frame cannot be selected. That gap is coverage error.
• Inclusion probability (written π, pi, for person i: π_i). The chance that person i ends up in the sample under the design.
• Design weight. The inverse of the inclusion probability, w_i = 1/π_i. A person selected with a 1 in 500 chance stands in for 500 people.
• Simple random sample (SRS). Every possible sample of size n has the same chance of selection.
• Stratification. Dividing the frame into groups (strata) and sampling separately within each.
• Cluster sampling. Selecting groups (blocks, schools, precincts) first, then people inside them.
• Design effect (deff). The variance of an estimate under the actual design divided by the variance it would have under a simple random sample of the same size.
• Effective sample size (n_e). The sample size divided by the design effect: the size of a simple random sample that would give the same precision.⁠⁵
• Response rate. A standardized ratio of completed interviews to eligible sampled cases, defined in several variants by AAPOR.⁠³

Evidence

Inference comes from the selection mechanism

The central idea of design-based survey inference is that randomness is introduced by the researcher, so its properties are known. Horvitz and Thompson’s 1952 paper in the Journal of the American Statistical Association presented “a general technique for the treatment of samples drawn without replacement from finite universes when unequal selection probabilities are used”.⁠¹ The estimator that carries their names weights each observed value by the inverse of its inclusion probability. In plain text, for a population total of some quantity y:

Y_hat = sum over sampled i of ( y_i / pi_i )

If every π_i is known and greater than zero, this estimator is unbiased for the population total over repeated samples. Nothing in that statement depends on the sample being large. It depends entirely on knowing π_i. That is the line between a probability sample and an opt-in sample, and it is why the size of an open participation exercise does not, by itself, license any statement about a wider public.

The history of polling supplies the standard cautionary case. As Wang, Rothschild, Goel and Gelman recount, the Literary Digest in 1936 “conducted a mail-in survey that attracted over two million responses” and predicted a landslide for Alf Landon, while Franklin Roosevelt “won the election decisively, carrying every state except for Maine and Vermont”.⁠² The same authors note that pollsters including George Gallup, Archibald Crossley and Elmo Roper “used considerably smaller but representative samples, and predicted the election outcome with a reasonable level of accuracy”.⁠²

The arithmetic of sampling error

For a proportion p estimated from a simple random sample of size n drawn from a population of size N, the standard error is:

SE(p_hat) = sqrt( p(1-p) / n ) * sqrt( 1-n/N )

The second factor, the finite population correction, is close to 1 whenever the sample is a small fraction of the population. That is why a national survey of 1,000 and a city survey of 1,000 have nearly the same sampling error. At p = 0.5, the most conservative case, a 95% margin of error (1.96 times the standard error) is about plus or minus 3.1 points for n = 1,000 and about plus or minus 4.9 points for n = 400. These figures are arithmetic, not findings, and they describe only random sampling error under a simple random sample. They say nothing about coverage error, nonresponse, or question wording, the other error sources discussed in Question design.

Stratification and allocation

Stratified sampling has been a core technique since at least Jerzy Neyman’s 1934 paper in the Journal of the Royal Statistical Society, titled “On the Two Different Aspects of the Representative Method: The Method of Stratified Sampling and the Method of Purposive Selection”.⁠⁴ Stratification can only reduce sampling variance relative to an SRS of the same size when strata differ on the measured variable, because it removes the chance of an unlucky mix across strata.

How many cases go into each stratum is a design choice. Proportional allocation gives each stratum h a sample share equal to its population share N_h/N. The allocation that minimizes variance for a fixed total sample, usually called Neyman allocation, sets:

n_h proportional to N_h * S_h

where S_h is the standard deviation of the variable within stratum h. Strata that are larger or more internally varied get more cases.

Many civic surveys deliberately depart from both, because they want reliable estimates for every subgroup. The Survey of the Performance of American Elections (SPAE), which studies how voters experience voting, interviews “a total of 200 registered voters in each of the 50 states (as well as Washington, DC),” a design the MIT Election Data and Science Lab says “makes it possible to paint a more accurate portrait of voters’ experience in each state” and to compare states.⁠⁶ The cost of equal allocation is that national estimates need weights, because a small state’s 200 voters stand for far fewer people than a large state’s 200.

Design effects and effective sample size

Unequal weights and clustering both inflate variance. The National Center for Health Statistics (NCHS) defines the effective sample size for a complex survey as “the sample size, n, divided by the design effect”:⁠⁵

n_e = n / deff, where deff = Var_design(p_hat) / ( p(1-p)/n )

Two widely used approximations show where design effects come from. They are standard textbook results, stated here as working formulas rather than findings:

• From unequal weights: deff_w is approximately 1 + CV(w)^2, where CV(w) is the coefficient of variation of the weights. Equivalently, n_e is approximately ( sum w_i )^2 / sum( w_i^2 ).
• From clustering: deff_c is approximately 1 + (m-1) * rho, where m is the average number of interviews per cluster and rho is the intraclass correlation, the degree to which people in the same cluster answer alike.

Hypothetical example. A design takes 200 interviews in each of 51 jurisdictions (10,200 total), and national estimates weight each jurisdiction by its population share s_h. Because each jurisdiction has the same n, the unequal-weighting design effect reduces to deff_w = 51 * sum( s_h^2 ). If population shares were all equal, deff_w would be 1. The more unequal they are, the larger it becomes: if the sum of squared shares were 0.04, deff_w would be about 2.0, and the 10,200 interviews would have the national precision of roughly 5,100 simple random interviews. The design trades national precision for jurisdiction-level reliability, which is the right trade if jurisdiction comparisons are the purpose.

NCHS turns effective sample size into a publication rule: proportions should rest on “a minimum denominator sample size and effective denominator sample size (when applicable) of 30,” with further rules on confidence interval width.⁠⁵ Those rules are discussed in Small-group suppression.

Frames: who can be selected at all

A probability sample is only as good as its frame. The American Community Survey (ACS), the Census Bureau’s largest household survey, lists 3,543,672 initial addresses for 2024, with 1,944,002 final housing unit interviews.⁠⁷ Even at that scale the Census Bureau publishes 1-year estimates only “for areas with populations of 65,000+” and relies on 5-year pooled estimates to cover “all areas”.⁠⁸ Small places need years of accumulated sample to support stable figures. That is a direct lesson for any civic exercise that wants to report results for a small town or a single precinct.

Pew Research Center’s American Trends Panel shows how a modern probability panel builds its frame. Pew describes the panel as a “multimode, probability-based survey panel” of “roughly 10,000 adults.” Its first recruitments in 2014, 2015 and 2017 used random telephone calls, and “in 2018, the ATP switched from telephone to address-based sampling (ABS) recruitment,” drawing households from the U.S. Postal Service address list.⁠⁹ People without internet were handled differently over time: mailed paper questionnaires at first, then tablets and data plans, and from mid-2024 a choice of online or phone, which Pew says “ensures that U.S. adults who don’t use the internet are represented on the panel”.⁠⁹ Coverage is engineered, not assumed.

Response rates have formulas, and the formulas matter

AAPOR’s Standard Definitions (tenth edition, 2023) standardizes how cases are coded and how outcome rates are computed.⁠³ Using I for complete interviews, P for partials, R for refusals and breakoffs, NC for non-contacts, O for other non-interviews, UH and UO for cases of unknown eligibility, and e for the estimated share of unknown cases that are eligible:

• RR1 = I / ( (I+P) + (R+NC+O) + (UH+UO) ), which AAPOR calls “the minimum response rate.”
• RR3 = I / ( (I+P) + (R+NC+O) + e(UH+UO) ), which AAPOR says “can be considered the most-common AAPOR response rate in reporting.”
• RR6 counts partials as respondents and assumes no unknown cases are eligible; AAPOR says it “represents the maximum response rate”.⁠³

The choice of e is where a response rate can be inflated. AAPOR’s rule is blunt: “One must not select a proportion to boost the response rate,” and “the basis for the estimate must be explicitly stated and detailed”.⁠³ For panels, AAPOR requires a cumulative rate that multiplies recruitment, profile and completion stages: CUMRR = RECR * PROR * COMR.⁠³ A panel with 10% recruitment and 80% completion has a cumulative rate of at most 8%, not 80%.

Response rates have fallen. Wang and colleagues, citing Pew data, report that by one measure random digit dialing response rates fell “from 36% in 1997 to 9% in 2012”.⁠² Whether that decline produces bias is a separate question, treated in Measurement: a low rate raises the risk of bias but does not prove it.

Competing interpretations

“A probability sample with a 6% response rate is no longer a probability sample.” On this view, once most selected people decline, the respondents are self-selected in all but name, and weighting is doing the same modeling work it does for opt-in data. Pew’s 2023 comparison gives this view some support: all three probability-based panels it tested “overestimated 2020 turnout by 8 or 9 points,” while two opt-in samples came within 1 point on that item.⁠¹⁰

“Probability designs still earn their keep.” The same Pew study found an average absolute error of 2.6 points across 28 benchmarks for probability-based panels against 5.8 points for opt-in samples, and summarized that opt-in samples “are about half as accurate as probability-based panels”.⁠¹⁰ On this view, a known frame and a documented selection process still constrain error in ways volunteer samples cannot, even when response is low.

“Design-based and model-based inference are converging.” Elliott and Valliant describe design-based inference as relying on “the random mechanism used by the sampler,” and contrast it with approaches that estimate pseudo-inclusion probabilities or model the outcome for nonsampled units.⁠¹¹ Many practitioners now combine both. Critics respond that a model is only as good as its assumptions, and assumptions about who did not answer are the hardest to check.

“Sample size is what the public understands.” Large numbers are persuasive. The Literary Digest case.⁠² is the standing counterexample: two million self-selected responses lost to a few thousand selected ones.

Application to this framework

The following are design proposals for My Vote Forward, not results.

1. Call open participation what it is. A priority exercise anyone can join has no inclusion probabilities. Its releases describe participants and do not carry a margin of error. Using a margin of error for such data would claim a precision that no selection mechanism supports.
2. Separate the participation channel from any measurement channel. If an institution wants to know how a whole population ranks priorities, the defensible tool is a separate probability survey with a documented frame, response rate (stating which AAPOR rate and the value of e), weights and design effect. That survey could then be compared with open participation to show where participants differ. Who would fund and govern such a survey is an open question, listed below.
3. Publish effective sample sizes, not only raw counts. Where any weighting is applied, releases would report n, the design effect and n_e for each published figure, following the NCHS definition⁠⁵
4. Plan geography from the start. The ACS’s 65,000 threshold for 1-year estimates⁠⁸ shows how much sample small areas need. A framework that reports by district must either accept wider uncertainty, pool across time with that pooling disclosed, or not report.
5. Use stratification honestly. If a companion survey oversamples small communities so their priorities can be reported, the release must say so and show both weighted and unweighted figures, so readers can see the design rather than infer it.

Hypothetical example. A county commissions a 1,000-person address-based survey alongside an open priority exercise with 6,000 participants. The survey’s weighted estimate that 41% place road maintenance in their top three carries a margin of error computed from n_e, not n. The open exercise reports that 58% of its 6,000 participants did the same, with no margin of error and a note that participants were self-selected. The gap between 41% and 58% is itself informative: it shows how participants differ from the sampled public on this item.

Unresolved questions

• Who should sponsor and govern a probability companion survey so that it is independent of the platform it evaluates? See governance.
• What minimum effective sample size should a district-level priority figure require, and should the rule follow NCHS’s 30⁠⁵ or something stricter for ranked data?
• How should a release combine a small, high-quality probability sample with a large open exercise? Meng’s work, discussed in Opt-in panels, suggests the small sample deserves far more weight than its size implies.
• Can address-based recruitment reach people who move often, live in group quarters, or lack a stable address, and what is lost if it cannot?
• How should a framework report results when the design effect itself is uncertain because weights depend on estimated population totals?